---
date: 2019-09-27
authors: jnm2377
---

# `v10.6.3` and `v10.6.4` broken patch release

**Summary**

Different new feature related PRs were making their way into the patch release
PR. These releases were done manually, as well, but there was no way of knowing
when PRs were tied to a new feature. `v10.6.3` was released under `next` after
successfully opening a PR for it, with ci passing build. It wasn't until testing
`next` that we realized it was actually broken because of a new token. We then
cut `v10.6.4` to fix it, and cherry-picked the PRs manually again. We ended up
having to re-do this process 4 times because each time, different PRs were being
tied to a new feature. The commit titles are not enough for an endgame developer
to know that the fixes are feat related. When `v10.6.4` was finally published
under next, everything was working as expected. However, when updating to latest
our packages were broken for a while since the endgame doc script was not
correct.

**Impact**

The impact of this is that it is time consuming for the endgame developer, who
has regular project work to do on top of endgame work. All in all, it took a
whole day to release a patch. The other impact is that our users can run into
issues installing our packages.

**Root causes**:

The main reason for this problem is that there is no current convention to
specify that a `chore`, `refactor`, `docs` or `fix` PR is related to a `feat`
PR.

The other root cause of one of the problems comes from a typo in our docs, which
meant the script we ran to update `latest` was wrong.

**Detection**

We discovered that new features were included in the patch by testing our `next`
tag and realizing it was broken, or seeing the ci build fail in the patch
release PRs.

We discovered that the `latest` tag was then broken through a user who opened an
issue saying they couldn't add our package as a dependency.

**Resolution**

We addressed the new features in the patch by releasing a new patch that
excluded the original PR that was breaking the patch. This new patch however had
to be re-released 4 times because each time it was discovered that a different
PR had a new feature. Since we hadn't published to `next` the resolution was to
keep re-releasing and opening a PR until we saw that the ci build was passing,
which meant there were no new features.

We addressed the broken `latest` tag by running the correct script to update the
tag.

**Action Items**

| Action item                           | Owner      | Bug                                                        |
| ------------------------------------- | ---------- | ---------------------------------------------------------- |
| Update package versions for `v10.6.3` | @jnm2377   | https://github.com/carbon-design-system/carbon/pull/4075   |
| Update package versions for `v10.6.4` | @jnm2377   | https://github.com/carbon-design-system/carbon/pull/4108   |
| Update package versions for `v10.6.4` | @jnm2377   | https://github.com/carbon-design-system/carbon/pull/4111   |
| Update package versions for `v10.6.4` | @jnm2377   | https://github.com/carbon-design-system/carbon/pull/4112   |
| Re-run update `latest` script         | @joshblack | https://github.com/carbon-design-system/carbon/issues/4115 |

## Lessons learned

**What went well**

- We were able to notice that `v10.6.3` was broken once we tested `next`
- We were able to fix the updated packages quickly once we realized the `latest`
  tag was broken after publishing `v10.6.4`

**What went wrong**

- `v10.6.3` was totally broken and it went undetected before publishing to
  `next` since ci build passed for it
- When manually cherry-picking PRs for the patch release, there was no way of
  knowing that so many of them had new features
- The script for publishing to `latest` had a typo which broke the updated
  packages

**Where we got lucky**

- We publish all of these under `next` to test before publishing to `latest`
- The `latest` tag wasn't broken for a long time and we were able to quickly
  resolve it after the initial inquiry

## Timeline

2019-09-25 **(all times in UTC)**

- Test out `v10.6.3` patch under the `next` tag
- Realize it was broken
- Re-run release steps to update the `next` tag

2019-09-26 **(all times in UTC)**

- 10:00 Test out re-do of `v10.6.3`
- 10:20 Second attempt to fix broken patch
- 10:40 Another test of `v10.6.3`
- 11:30 Reach out to @joshblack about how to fix broken patch
- 11:45 Go through patch release steps for `v10.6.4`
- 11:55 Open first `v10.6.4` PR
- 13:00 Go through patch release steps for second time for `v10.6.4`
- 13:15 Open second `v10.6.4` PR
- 14:00 Go through patch release steps for `v10.6.4` for the third time
- 14:15 Open third `v10.6.4` PR
- 14:30 Release `v10.6.4` under `next`
- 14:45 Tested the `next` tag
- 15:00 Published `v10.6.4` under the `latest` tag
- 16:30 @asudoh inquired about a broken publish in our dev channel (related to
  [#4115](https://github.com/carbon-design-system/carbon/issues/4115))
- 16:40 @joshblack re-ran the publishing scrips to fix the updated packages

## Supporting information

<!-- Any additional information that you might reference earlier on in the
postmortem -->
