Sync .github directory with azure-sdk-tools for PR 15210#58397
Open
Sync .github directory with azure-sdk-tools for PR 15210#58397
Conversation
Re-creates PR #15062 on a fresh branch off current upstream/main to avoid sync-pipeline CI issues on the original branch. Original branch squad/15049-sync-shared-skills and PR #15062 are preserved unchanged for comparison. Makes skills the single source of truth by consolidating instruction file content into the corresponding skill SKILL.md files, then deleting the redundant instruction files. Also renames 5 shared skill directories with the 'azsdk-common-' prefix required by the sync pipeline. 7 instruction files deleted (content merged into skills): - local-sdk-workflow.instructions.md -> azsdk-common-generate-sdk-locally - create-release-plan.instructions.md -> azsdk-common-prepare-release-plan - sdk-details-in-release-plan.instructions.md -> azsdk-common-prepare-release-plan - verify-namespace-approval.instructions.md -> azsdk-common-prepare-release-plan - check-package-readiness.instructions.md -> azsdk-common-sdk-release - check-package-validation.instructions.md -> azsdk-common-generate-sdk-locally - copilot/sdk-release.instructions.md -> azsdk-common-sdk-release 3 skills enriched with details from instruction files: - generate-sdk-locally: commit checkpoints, language selection, config file identification, SDK project path, metadata update tools, validation params - prepare-release-plan: critical LLM instructions (NextSteps field), Release Planner Tool link, per-language package name extraction rules, valid/invalid examples table, JSON structure, namespace approval flow - sdk-release: readiness check details, interaction flow, do-not-ask-for-PR guard, case-sensitive language note 5 shared skill directories renamed with 'azsdk-common-' prefix: apiview-feedback-resolution, generate-sdk-locally, pipeline-troubleshooting, prepare-release-plan, sdk-release typespec-to-sdk orchestrator updated to reference skills directly. Removed all related_skill frontmatter and blockquotes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lease-plan; only change is name prefix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…efix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
azure-sdk
requested review from
ArthurMa1978,
jsquire and
ronniegeraghty
as code owners
azure-sdk
added
EngSys
Contributor
There was a problem hiding this comment.
Pull request overview
This PR syncs the .github “skills” definitions from azure-sdk-tools PR 15210 into this repo, adding/refreshing skill documentation plus task/eval YAML used to validate triggering behavior and tool usage for common SDK engineering workflows.
Changes:
- Adds new “common” skills and reference material for SDK release, release-plan preparation, pipeline troubleshooting, local generation, and APIView feedback resolution.
- Adds task YAMLs for basic/edge/negative scenarios that assert expected outputs and tool calls.
- Adds eval suites (
eval.yaml) plus trigger tests and eval task sets (evals/*) for the new skills.
Reviewed changes
Copilot reviewed 64 out of 64 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/azsdk-common-sdk-release/tasks/release-trigger-001.yaml | Adds task asserting release trigger flow calls azsdk_release_sdk. |
| .github/skills/azsdk-common-sdk-release/tasks/release-readiness-001.yaml | Adds readiness-only task asserting azsdk_release_sdk usage. |
| .github/skills/azsdk-common-sdk-release/tasks/release-negative-001.yaml | Adds negative task for unrelated request (CODEOWNERS). |
| .github/skills/azsdk-common-sdk-release/tasks/release-basic-001.yaml | Adds happy-path task for full release flow guidance. |
| .github/skills/azsdk-common-sdk-release/evals/trigger_tests.yaml | Adds prompt trigger/anti-trigger test set for skill activation. |
| .github/skills/azsdk-common-sdk-release/evals/tasks/basic-trigger.yaml | Adds eval task expecting skill activation on release prompts. |
| .github/skills/azsdk-common-sdk-release/evals/tasks/anti-trigger.yaml | Adds eval task expecting no activation on unrelated prompts. |
| .github/skills/azsdk-common-sdk-release/evals/eval.yaml | Adds eval-runner config for the evals/ suite. |
| .github/skills/azsdk-common-sdk-release/eval.yaml | Adds top-level eval suite config for the skill. |
| .github/skills/azsdk-common-sdk-release/SKILL.md | Adds skill definition and step-by-step guidance for release readiness + triggering. |
| .github/skills/azsdk-common-prepare-release-plan/tasks/should-not-trigger.yaml | Adds negative task ensuring package-release requests don’t activate this skill. |
| .github/skills/azsdk-common-prepare-release-plan/tasks/link-sdk-prs.yaml | Adds task for linking SDK PRs to an existing release plan. |
| .github/skills/azsdk-common-prepare-release-plan/tasks/edge-case.yaml | Adds validation/auto-correction edge-case task for package name formats. |
| .github/skills/azsdk-common-prepare-release-plan/tasks/basic-usage.yaml | Adds basic task for creating a new release plan from a spec PR. |
| .github/skills/azsdk-common-prepare-release-plan/references/release-plan-details.md | Adds detailed reference steps and constraints for release-plan handling. |
| .github/skills/azsdk-common-prepare-release-plan/fixtures/tspconfig.yaml | Adds fixture TypeSpec config used in tests/examples. |
| .github/skills/azsdk-common-prepare-release-plan/evals/trigger_tests.yaml | Adds activation trigger/anti-trigger prompts for the skill. |
| .github/skills/azsdk-common-prepare-release-plan/evals/tasks/basic-trigger.yaml | Adds eval task expecting activation on release-plan prompts. |
| .github/skills/azsdk-common-prepare-release-plan/evals/tasks/anti-trigger.yaml | Adds eval task expecting non-activation on unrelated prompts. |
| .github/skills/azsdk-common-prepare-release-plan/evals/eval.yaml | Adds eval-runner config for the evals/ suite. |
| .github/skills/azsdk-common-prepare-release-plan/eval.yaml | Adds top-level eval suite config for the skill. |
| .github/skills/azsdk-common-prepare-release-plan/SKILL.md | Adds skill definition and steps for creating/updating/linking release plans. |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/should-not-trigger.yaml | Adds negative task to avoid activation for TypeSpec authoring prompts. |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/local-reproduction.yaml | Adds task guiding local repro of CI failures. |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/edge-case.yaml | Adds multi-language failure triage task. |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/basic-usage.yaml | Adds basic task that calls azsdk_analyze_pipeline for a pipeline URL. |
| .github/skills/azsdk-common-pipeline-troubleshooting/references/failure-patterns.md | Adds reference doc of failure patterns and remediations. |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/trigger_tests.yaml | Adds activation trigger/anti-trigger prompts for pipeline troubleshooting. |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/tasks/basic-trigger.yaml | Adds eval task expecting activation on pipeline-failing prompts. |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/tasks/anti-trigger.yaml | Adds eval task expecting non-activation on unrelated prompts. |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/eval.yaml | Adds eval-runner config for the evals/ suite. |
| .github/skills/azsdk-common-pipeline-troubleshooting/eval.yaml | Adds top-level eval suite config for the skill. |
| .github/skills/azsdk-common-pipeline-troubleshooting/SKILL.md | Adds skill definition and troubleshooting workflow tied to MCP tools. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/update-version.yaml | Adds task asserting version update tool call. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/update-metadata.yaml | Adds task asserting metadata + ci.yml update tool call. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/update-changelog.yaml | Adds task asserting changelog update tool call. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/rename-client.yaml | Adds customization task for renaming client/model via TypeSpec. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/hide-operation.yaml | Adds customization task for hiding internal operations from SDK APIs. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/full-workflow.yaml | Adds end-to-end local generation/build/check/test/metadata workflow task. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/edge-case.yaml | Adds edge-case task for build-failure recovery via customization. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/customization-workflow.yaml | Adds task for resolving duplicate-field conflicts after regen. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/breaking-changes.yaml | Adds task for resolving breaking-change failures via customization. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/basic-usage.yaml | Adds basic task for local SDK generation flow starting with setup verification. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/anti-trigger.yaml | Adds anti-trigger task to avoid activation for pipeline-based generation. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/analyzer-errors.yaml | Adds task for handling .NET analyzer errors via customization. |
| .github/skills/azsdk-common-generate-sdk-locally/references/sdk-repos.md | Adds language repo mapping and tool reference for local generation workflows. |
| .github/skills/azsdk-common-generate-sdk-locally/references/detailed-workflow.md | Adds detailed guidance for config identification and commit checkpoints. |
| .github/skills/azsdk-common-generate-sdk-locally/references/customization-workflow.md | Adds detailed 2-phase customization workflow reference. |
| .github/skills/azsdk-common-generate-sdk-locally/fixtures/tspconfig.yaml | Adds fixture TypeSpec config for generate-locally scenarios. |
| .github/skills/azsdk-common-generate-sdk-locally/eval.yaml | Adds top-level eval suite config for the skill. |
| .github/skills/azsdk-common-generate-sdk-locally/SKILL.md | Adds skill definition and step-by-step local generation guidance. |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/should-not-trigger.yaml | Adds negative task to avoid activation for creating TypeSpec projects. |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/no-feedback.yaml | Adds task handling “no APIView comments found” scenario. |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/edge-case.yaml | Adds edge-case task for APIView feedback requiring TypeSpec changes. |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/basic-usage.yaml | Adds basic task that calls azsdk_apiview_get_comments. |
| .github/skills/azsdk-common-apiview-feedback-resolution/references/feedback-resolution-steps.md | Adds detailed reference steps for categorizing/resolving APIView feedback. |
| .github/skills/azsdk-common-apiview-feedback-resolution/fixtures/apiview-comment.json | Adds fixture APIView comment payload used for testing/examples. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger_tests.yaml | Adds activation trigger/anti-trigger prompts for APIView skill. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/tasks/url-trigger.yaml | Adds eval task for URL-only prompts to trigger the skill. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/tasks/basic-trigger.yaml | Adds eval task expecting activation on explicit APIView prompts. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/tasks/anti-trigger.yaml | Adds eval task expecting non-activation on unrelated prompts. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml | Adds eval-runner config for the evals/ suite. |
| .github/skills/azsdk-common-apiview-feedback-resolution/eval.yaml | Adds top-level eval suite config for the skill. |
| .github/skills/azsdk-common-apiview-feedback-resolution/SKILL.md | Adds skill definition and workflow for retrieving + resolving APIView feedback. |
Comment on lines
+21
to
+27
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
graders: is followed by a list item at the top indentation level (- type: text) and regex_not_match: is followed by unindented list items. As written, this YAML is invalid (the list items are not nested under their keys). Indent the graders list under graders: and indent the regex patterns under regex_not_match: so the file parses correctly.
Suggested change
| - type: text | |
| name: no_fatal_errors | |
| config: | |
| regex_not_match: | |
| - (?i)fatal error | |
| - (?i)unhandled exception | |
| - '(?i)panic:' | |
| - type: text | |
| name: no_fatal_errors | |
| config: | |
| regex_not_match: | |
| - (?i)fatal error | |
| - (?i)unhandled exception | |
| - '(?i)panic:' |
Comment on lines
+3
to
+14
| description: | | ||
| Test that the skill does NOT trigger for non-release requests. | ||
| tags: | ||
| - negative | ||
| inputs: | ||
| prompt: "I need to validate my CODEOWNERS file for the Python SDK repo." | ||
| expected: | ||
| output_contains: | ||
| - "CODEOWNERS" | ||
| behavior: | ||
| max_tool_calls: 2 | ||
| max_response_time_ms: 30000 |
There was a problem hiding this comment.
This task is described as ensuring the skill does NOT trigger for non-release requests, but the assertions only require CODEOWNERS to appear and allow up to 2 tool calls. This doesn't actually verify non-triggering (the skill could still call azsdk_release_sdk). Add negative assertions (e.g., output_not_contains azsdk_release_sdk/"release") and/or require zero tool calls for this task.
| config: | ||
| regex_not_match: | ||
| - (?i)error occurred | ||
| - (?i)failed |
There was a problem hiding this comment.
The no_fatal_errors grader forbids any case-insensitive match of failed, which is likely to appear in normal guidance for this skill (many tasks involve build/test failures). This will make the eval suite overly brittle and may fail valid responses. Consider narrowing this to truly fatal/error signatures (e.g., "fatal error", "unhandled exception", "panic:") or more specific failure phrases returned by the executor/tooling rather than the generic word "failed".
Suggested change
| - (?i)failed | |
| - (?i)fatal error | |
| - (?i)unhandled exception | |
| - (?i)panic: |
Comment on lines
+20
to
+27
| graders: | ||
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
graders: is followed by a list item at the top indentation level (- type: text) and regex_not_match: is followed by unindented list items. As written, this YAML is invalid (the list items are not nested under their keys). Indent the graders list under graders: and indent the regex patterns under regex_not_match: so the file parses correctly.
Comment on lines
+20
to
+27
| graders: | ||
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
graders: is followed by a list item at the top indentation level (- type: text) and regex_not_match: is followed by unindented list items. As written, this YAML is invalid (the list items are not nested under their keys). Indent the graders list under graders: and indent the regex patterns under regex_not_match: so the file parses correctly.
Comment on lines
+20
to
+27
| graders: | ||
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
graders: is followed by a list item at the top indentation level (- type: text) and regex_not_match: is followed by unindented list items. As written, this YAML is invalid (the list items are not nested under their keys). Indent the graders list under graders: and indent the regex patterns under regex_not_match: so the file parses correctly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync .github directory with azure-sdk-tools for PR Azure/azure-sdk-tools#15210 See eng/common workflow