# 06: Callbacks And Cleanup ## Outcome Observe page `on_action`, target `before_run`, target `after_run`, retained source `on_deactivate`, fixed replacement order, and one retry of the exact failed cleanup snapshot. ## Prerequisites - Complete Tutorials 01 through 05. - Permission to create lesson objects, run Apps, inspect transitions, and retry cleanup for your action. - About 35 minutes. This lesson uses no cloud service or secret. ## Files - `Dockerfile` creates the callback image and installs `run.sh`. - `run.sh` emits one bounded marker for each callback. It uses `/lemc/private/tutorial-06` only for recipe activation and retry markers. - `cookbook.yaml` declares one page hook and recipe-owned hooks on three recipes. Its eight executable coordinates need separate managed image bindings. Read the callback owners before you run the lesson: | Callback or phase | Owner | Purpose | | --- | --- | --- | | Page `on_action` | The accepted page action | Record that action admission completed. | | Source `on_deactivate` | The active source recipe generation | Remove state that the replaced generation owns. | | Target `before_run` | The selected target recipe | Prepare target work after source cleanup succeeds. | | Target `now` | The selected target recipe | Run the selected recipe work. | | Target `after_run` | The selected target recipe | Verify target work after NOW succeeds. | The private marker files demonstrate lifecycle ownership. They are not public artifacts. The retryable cleanup fails once, retains its exact source snapshot, and succeeds only when LEMC retries that failed transition. ## Web 1. Review `cookbook.yaml` beside `run.sh`. Count eight executable coordinates: page `on_action`; Environment A `on_deactivate` and `now`; Environment B `before_run`, `now`, and `after_run`; and Retryable Cleanup `on_deactivate` and `now`. 2. Build `tutorial-callbacks` from `Dockerfile` and `run.sh`. Wait for the Builder execution to reach terminal success and record the verified build identity. 3. Create **Tutorial 06 Callbacks And Cleanup** from `cookbook.yaml`. Bind the verified build to all eight coordinates. Confirm that each server copy uses the same managed digest. Publish the Cookbook and create a private test App. 4. Run **Environment A**. Record the transition ID and its child task IDs. Confirm that page `on_action` reports `accepted`, then Environment A NOW reports `completed`. Environment A becomes the active source generation. 5. Run **Environment B** with a new action request. Inspect the child list and event order. It must be page `on_action`, Environment A `on_deactivate`, Environment B `before_run`, Environment B NOW, and Environment B `after_run`. Confirm that all five children reach terminal success. 6. Run **Retryable Cleanup** with a new action request. Wait for its transition to succeed. It is now the active source generation and owns its retained `on_deactivate` snapshot. 7. Run **Environment B** again. Record this replacement transition. Page `on_action` can succeed, but Retryable Cleanup `on_deactivate` exits with code 42 on its first attempt. Confirm the `cleanup_failed` state. Confirm that no Environment B target child starts after that failure. 8. Select **Retry cleanup** on the failed replacement transition. Confirm that the retry uses the same source cleanup snapshot and reports `retried-and-removed`. It must then continue to Environment B `before_run`, NOW, and `after_run`. It must not repeat the successful page `on_action` child. 9. Inspect the final transition inventory. Every child must be terminal, Environment B must be active, and no Retryable Cleanup marker can remain active. The target's `on_deactivate` never runs during its own activation. LEMC retains it until a later accepted action replaces that exact generation. ## LEMCSSH Use `jobs surface` to inspect the value-free effect order before submission. Follow each returned transition instead of guessing a child task. Save the transition ID from the failed Retryable-Cleanup-to-B replacement. That is the only valid ID for the cleanup retry below. ```sh ssh LEMC_ALIAS jobs surface app --uuid APP_UUID \ --scope individual --format=json ssh LEMC_ALIAS jobs run app --uuid APP_UUID --page 1 \ --scope individual --recipe environment-a \ --idempotency-key REQUEST_UUID --format=json ssh LEMC_ALIAS jobs run app --uuid APP_UUID --page 1 \ --scope individual --recipe environment-b \ --idempotency-key REQUEST_UUID --format=json ssh LEMC_ALIAS jobs follow --transition-id TRANSITION_ID --render plain ssh LEMC_ALIAS jobs run app --uuid APP_UUID --page 1 \ --scope individual --recipe retryable-cleanup \ --idempotency-key REQUEST_UUID --format=json ssh LEMC_ALIAS jobs run app --uuid APP_UUID --page 1 \ --scope individual --recipe environment-b \ --idempotency-key REQUEST_UUID --format=json ssh LEMC_ALIAS jobs retry-cleanup --transition-id TRANSITION_ID --format=json ssh LEMC_ALIAS jobs status app --uuid APP_UUID --page 1 \ --scope individual --transition-id TRANSITION_ID --format=json ``` Each intended recipe action needs a new request UUID. A transport replay of the same action uses its original UUID and must not add callback children. The retry command takes the failed replacement transition ID. Do not pass the successful Retryable Cleanup activation transition or one of its child task IDs. ## Expected result The successful A-to-B transition has this order: ```text on_action -> on_deactivate -> before_run -> now -> after_run ``` Keep the transition and marker evidence together: | Check | Expected evidence | | --- | --- | | A activation | `on_action accepted`, then Environment A `now completed` | | A-to-B replacement | A `on_deactivate removed` occurs before B `before_run prepared` | | B target work | `before_run prepared`, `now completed`, then `after_run verified` | | First retryable cleanup | Exit code 42 and transition state `cleanup_failed` | | Failure boundary | No B target child starts after the failed cleanup child | | Cleanup retry | The same cleanup snapshot reports `retried-and-removed` | | Resume boundary | B target work continues without a second successful page hook | The final replacement reaches terminal success. The failed cleanup attempt and its successful retry remain visible as parts of the same retained transition. ## Cleanup and retry Finish with **Environment B** active so the retryable marker is removed. Retry only the transition that reports failed cleanup. Read status first if the retry request has an unknown transport result. A repeated request must refer to the same retained snapshot and must not add a second cleanup child. Confirm that all transition children are terminal. Then remove the App and archive the Cookbook and Development Image. Never edit the Cookbook to replace cleanup code while an old activation is held. Repair access to the retained digest if required, then retry the exact snapshot. Do not delete private state or Runner data as a cleanup substitute. ## Security and cost Callbacks are hosted recipe jobs, not Web-process code. Whole-transition preflight resolves every image before source cleanup begins. The lesson's `/lemc/private` markers stay inside the task-private contract and are not downloadable artifacts. Callback metadata and output must not contain form values, secret values, registry credentials, or raw idempotency keys. This lesson has no external cost. ## Troubleshooting - A missing callback child usually means the action failed authorization, validation, image preflight, or idempotency admission before acceptance. - `cleanup_failed` is terminal for that attempt. Use the exact transition's retry action; do not submit a new target action as a cleanup substitute. - An image error on one hook means that callback coordinate was not bound. Page and recipe hooks do not share an implicit binding. - A repeated successful page hook after transport replay indicates an idempotency defect. Stop and preserve the transition inventory. - A retry that reports no failed cleanup usually used the wrong transition ID. Select the replacement transition that is in `cleanup_failed`. - A target child that starts before source cleanup succeeds violates the fixed replacement order. Stop new actions and preserve all child task IDs and event IDs. - A terminal successful retry with an active Retryable Cleanup marker is inconsistent with cleanup completion. Preserve the transition and private synchronization evidence. Do not remove the marker by host access. ## Internal validation Maintainers can run `./scripts/validate.sh` and the focused private callback tests. No public step requires a native client.