CAGED / DataEmpregos

DataEmpregos makes Brazil's monthly formal-employment movements easier to explore. CAGED prepares the underlying data by turning the official Novo CAGED source archives into geographic and occupational metrics that can be queried efficiently.

EventBridge Scheduler starts a Standard Step Functions workflow that checks for new archives, downloads required files to private S3 storage, and then runs a Fargate processing task. Step Functions owns sequencing, retries, and failure handling across the workflow.

The processor validates complete monthly file groups, aggregates records by geography and CBO occupational family, and writes precomputed metrics plus availability metadata to DynamoDB. The query path therefore reads a prepared data model instead of parsing source archives during a request.

CloudFront and AWS WAF form the public edge. Browser clients reach the DataEmpregos frontend, whose server boundary validates requests and calls a private API Gateway with its AWS identity; the API invokes a read-only query Lambda that retrieves the prepared DynamoDB metrics.

SNS, SQS, and a notifier Lambda decouple workflow notifications from ingestion. SQS retries transient delivery failures and isolates unresolved messages in a dead-letter queue, while structured CloudWatch logs and Step Functions execution history provide operational visibility.

Terraform manages the backend and frontend infrastructure in separate repositories. Each application repository owns its deployment pipeline: GitHub Actions assumes narrowly scoped AWS roles through OIDC to release Lambda ZIP artifacts, promote Lambda aliases, or build and deploy the Fargate processing image independently.