Skip to content

Certification Sync — Background ACA Job

Automatically pull certifications from Credly, Databricks (Accredible), and Microsoft Learn into the portfolio database, replacing the current static fallback entries with live badge data.

Goals

  • Remove manual maintenance of the certifications list
  • Show badge images, issue dates, and expiry dates on the frontend
  • Trigger a resync on demand from an admin endpoint
  • Run automatically on a schedule via an Azure Container Apps job

Source feasibility

Source Method Auth required Data quality
Credly Public JSON API — credly.com/users/sven-relijveld/badges.json None Full: name, issued/expiry dates, badge image URL, issuer
Databricks Accredible-hosted wallet page (credentials.databricks.com) None (page is public) Requires HTML scraping — JS-rendered SPA; need Playwright or Accredible API key from account settings
Microsoft Learn learn.microsoft.com/api/contentbrowser/search/credentials — undocumented public API discovered via exam-timeline None (User-Agent header only) Fields vary; returns credential type, name, issued date, status

Recommendation for v1: automate Credly and Microsoft Learn fully; attempt Databricks via HTML scrape with Playwright, fall back to manual if unreliable.

Data model changes

certifications table — new columns

Column Type Notes
source Text credly / databricks / microsoft / manual
external_id Text (nullable, unique per source) Credly badge ID, MS credential ID
badge_image_url Text (nullable) Remote image URL from source
issued_at TIMESTAMP WITH TIME ZONE (nullable)
expires_at TIMESTAMP WITH TIME ZONE (nullable)
last_synced_at TIMESTAMP WITH TIME ZONE (nullable) Set on every successful sync
sync_enabled Boolean Default true; set to false to pin a manual entry

Existing columns (title, description, icon_type, link, link_text, sort_order) are kept unchanged so the frontend needs no immediate changes.

Alembic migration

New file: api/alembic/versions/0008_add_certification_sync_fields.py

Backend — new files

api/services/credentials_sync.py

Three async fetch functions, one per source:

async def sync_credly() -> list[CertificationUpsert]: ...
async def sync_microsoft_learn() -> list[CertificationUpsert]: ...
async def sync_databricks() -> list[CertificationUpsert]: ...  # HTML scrape, may raise

Each returns a list of CertificationUpsert dataclass objects. A shared upsert_certifications(db, records) function matches on (source, external_id) and updates existing rows or inserts new ones. Records with sync_enabled=False are never touched.

Credly fetch:

GET https://www.credly.com/users/sven-relijveld/badges.json
# No auth. Returns data[].{badge_template.name, issued_at_date, expires_at_date,
#   badge_template.image.url, earner_path, state}
# Filter: state == "accepted" and public == True

Microsoft Learn fetch:

POST https://learn.microsoft.com/api/contentbrowser/search/credentials
Headers: {"User-Agent": "Mozilla/5.0 (compatible; PortfolioSync/1.0)"}
# Returns credentials with name, issued date, status, type

Databricks fetch (Accredible):

  • Attempt 1: check if credentials.databricks.com/profile/svenrelijveld92543/wallet exposes a JSON endpoint (.json suffix or XHR observed in browser devtools)
  • Attempt 2: lightweight Playwright scrape (headless Chromium) — parse credential cards
  • If neither works: log a warning and skip; records remain as manually-seeded entries

api/routers/sync.py

POST /api/admin/sync-certifications
  • Protected by EasyAuth (existing middleware — no additional auth code needed on prod; dev uses the same X-ZUMO-AUTH header bypass)
  • Calls all three sync functions, returns a summary: {credly: N, microsoft: N, databricks: N, errors: [...]}
  • Rate-limited to 10/hour

ACA background job

A separate Container App job (not a new image — reuses the existing backend image with a different entrypoint):

docker run portfolio-backend python -m api.jobs.sync_certifications

New file: api/jobs/sync_certifications.py — imports the sync service, runs all three sources, exits 0 on success / non-zero on failure (ACA job retries on non-zero exit).

Bicep changes

New resource in infra/modules/container_apps.bicep:

resource certSyncJob 'Microsoft.App/jobs@2023-05-01' = {
  name: '${prefix}-job-cert-sync'
  // schedule trigger — cron expression
  configuration: {
    triggerType: 'Schedule'
    scheduleTriggerConfig: {
      cronExpression: '0 6 * * 1'  // every Monday at 06:00 UTC
    }
    replicaTimeout: 300
    replicaRetryLimit: 2
  }
  // same image as backend, entrypoint override
  template: {
    containers: [{
      image: backendImage
      command: ['python', '-m', 'api.jobs.sync_certifications']
      env: backendEnvVars  // DATABASE_URL, APP_CONFIG_ENDPOINT, etc.
    }]
  }
}

On-demand trigger

az containerapp job start \
  --name dna-prd-portfolio-we-job-cert-sync \
  --resource-group dna-prd-portfolio-westeurope-rg

Also wired to the POST /api/admin/sync-certifications endpoint which calls the same job start via Azure SDK, so it can be triggered from the portfolio admin UI.

Frontend changes

Schema additions to Certification response

badgeImageUrl?: string    // null for manual entries without images
issuedAt?: string         // ISO date string
expiresAt?: string        // null if no expiry
source: string            // 'credly' | 'databricks' | 'microsoft' | 'manual'

CertificationsSection component updates

  • Show badge image (<img src={badgeImageUrl} />) when available, fall back to current icon
  • Show issued/expiry date below the title
  • Expired badges: dim the card (opacity-50) or show a "Expired" chip

Files to create / modify

File Action
api/alembic/versions/0008_add_certification_sync_fields.py Create — migration
api/db/models.py Modify — add columns to Certification
api/schemas/responses.py Modify — add fields to Certification schema
api/services/credentials_sync.py Create — fetch + upsert logic
api/jobs/sync_certifications.py Create — job entrypoint
api/routers/sync.py Create — admin trigger endpoint
api/main.py Modify — register sync router
api/repositories/certifications.py Modify — add upsert function
api/data/seed.json Modify — add source, external_id, sync_enabled to existing entries
infra/modules/container_apps.bicep Modify — add ACA job resource
client/src/components/certifications-section.tsx Modify — show badge image + dates
shared/api.types.ts Auto-regenerated via npm run types:generate after backend schema changes

Build sequence

  1. Migration + model + schema changes (backend only, no infra yet)
  2. Sync service + job entrypoint — testable locally via python -m api.jobs.sync_certifications
  3. Admin router + registration in main.py
  4. Frontend: update Certification schema display
  5. Bicep: add ACA job + deploy to dev, verify schedule trigger fires
  6. Seed existing manual entries with source=manual, sync_enabled=false
  7. Run first full sync, verify DB populated correctly
  8. Deploy to prod

Open questions

  • Databricks: check account settings at credentials.databricks.com for an Accredible API key. If available, use the official API instead of scraping.
  • Sort order: after sync, badges are ordered by issued_at DESC. Manual entries keep their existing sort_order. Decide if synced entries should interleave with manual ones or appear in a separate group.
  • Image caching: Credly image URLs are stable CDN URLs — safe to use directly. If hotlinking becomes unreliable, proxy through a backend /api/badge-image/{id} endpoint that caches to Azure Blob Storage.