Finding Leaked API Keys Before They Become Incidents

TL;DR: Secret scanning is useful only when discovery leads to fast remediation without creating another secret repository. GitMagnet monitors authorized public GitHub scopes, filters likely placeholders, validates candidates through provider-specific rules, and stores only a fingerprint plus masked fragments. The raw credential exists in memory long enough to classify the exposure, then disappears.

GitMagnet is a defensive exposure-monitoring project. It is intended for repositories and public scopes an operator is authorized to assess. A finding should trigger revocation, rotation, owner notification, and incident review, never use of the exposed credential.

A committed secret remains exposed after deletion

Removing an API key from the latest version of a file does not remove it from Git history, forks, caches, or any clone made before the cleanup. Prevention in CI is essential, but organizations also need to know what has already crossed the public boundary.

That creates a specific monitoring problem:

1. Discover repositories and revisions likely to contain provider credentials.
2. Separate plausible candidates from documentation examples and test fixtures.
3. Determine whether a candidate still represents a live exposure.
4. Retain enough evidence to deduplicate and remediate it.
5. Avoid turning the scanner database and logs into a second credential leak.

GitMagnet implements that lifecycle as a NestJS service with PostgreSQL persistence and a terminal operations interface.

Discovery uses two different budgets

GitHub does not expose one unlimited search primitive. Code-oriented and repository-oriented searches have different quotas and different strengths, so GitMagnet models two strategies.

Quick search starts from provider-aware patterns and targets likely code matches. Deep search starts from broader secret-related keywords, clones candidate repositories, scans eligible files, and inspects a bounded commit history. Search status records the current repository, file, commit, strategy, progress, and recent findings.

The rate-limit service tracks GitHub's core, search, and code-search resources independently. Work pauses when a required budget is exhausted and resumes after reset instead of hammering the API or failing an entire run. Quick search can switch to deep search when its quota or effectiveness falls below a useful threshold.

This is a better model than hiding quotas inside retry loops. Rate limits are part of the application's state and visible to the operator.

Pattern matching is only the first filter

A regular expression can identify the shape of a provider credential, but public repositories contain thousands of values deliberately designed to look like credentials: examples, test fixtures, placeholders, and documentation snippets.

GitMagnet associates each compiled pattern with a provider, then applies several additional checks:

• minimum body length after removing a known prefix;
• Shannon entropy over the variable portion;
• placeholder and repetitive-character detection;
• source-path filtering for examples, documentation, and tests;
• optional provider-specific context near the candidate;
• optional structural checks such as a Base64-compatible alphabet.

The output is a provider-labelled candidate, not a confirmed incident. Keeping those concepts separate prevents a regex match from being reported as a live credential.

Validation is provider-specific and bounded

Providers expose different safe ways to classify a credential. GitMagnet represents validation configuration in the provider domain rather than scattering endpoint assumptions through the scanner.

A validation request has a bounded timeout, provider-specific method and headers, retry behavior for transient failures, and an explicit success rule. Rate limiting and service unavailability do not silently become "invalid" results. That distinction matters during remediation: an unreachable provider is unknown, not safe.

The worker pool controls concurrent repository processing and validation. Tasks have priorities, observable lifecycle events, pause and resume controls, and a watchdog for work that exceeds its deadline. Worker count can be adjusted without changing the search domain.

This architecture improves throughput, but its more important property is backpressure. External quotas, network latency, and large repositories cannot create unbounded work inside the process.

The database never needs the raw credential

The safest stored secret is no secret at all.

When a candidate has been classified, GitMagnet computes a SHA-256 fingerprint and keeps only:

• the fingerprint used for deduplication;
• a short prefix and suffix for masked operator display;
• provider and repository identity;
• first-detected and last-validated timestamps;
• classification, risk, and remediation metadata.

The raw value is not written to PostgreSQL. Terminal output masks it, and persistence logs refer to the provider rather than printing the candidate. Valid and invalid findings use the same fingerprinting rule, preventing repeated validation from creating duplicate rows.

This is intentionally different from encrypting raw values at rest. Encryption would preserve the ability to recover a credential the service has no legitimate reason to use. A one-way identity is sufficient for exposure tracking.

DDD keeps providers replaceable

The project separates provider configuration, key discovery, extraction, validation, repository search, rate limiting, workers, and persistence. Domain objects validate provider patterns and credential identity; infrastructure repositories map them to TypeORM entities; controllers expose operations without owning scanning behavior.

The separation is most valuable when adding a provider. A provider contributes detection and validation configuration. It does not require another search pipeline or a fork of the worker system.

The test suite exercises domain invariants, extraction, entropy and placeholder filtering, provider validation, rate-limit behavior, worker concurrency, repository processing, persistence, security boundaries, and representative multi-provider workflows. Synthetic credentials and mocked provider endpoints keep tests from depending on real secrets.

Detection is the start of the incident workflow

A scanner cannot revoke a credential merely because it found one, and automatic use of a discovered credential would cross both an ethical and technical boundary. The correct next actions belong to an incident workflow:

confirmed exposure
  -> notify repository owner
  -> revoke credential
  -> issue replacement
  -> inspect provider audit logs
  -> remove secret from code and history where practical
  -> add prevention to local hooks and CI

GitMagnet currently provides discovery, classification, deduplication, and the operator surface. Organization ownership mapping, notification delivery, and provider-side revocation should be explicit integrations with their own authorization and audit controls.

The service is therefore not a credential collection tool. It is a way to reduce the time between a public mistake and a controlled response, while retaining less sensitive data than the problem it is designed to find.