Ever since we started shipping Acter to the Apple iOS AppStore, we wanted to have it on the Apple MacOS Store as well. With us building it on Rust and Flutter this should have been quite an easy feat as both have native support for MacOS. Yet actually shipping it was multiple months of try and error—with the last month spent on just a tiny problem caused by the Github Actions Runners. these are six niche tips we wished someone had told us before, that would have saved us months of work.

Nightly builds as the baseline

For testing and internal distribution we had nightly builds in Acter for a few months already. They would automatically be created every night at 3am (hence the name) if changes had been found on the main branch. Our build here consists of two parts: the internal Rust library and then we follow with a simple flutter build $target. So obviously, we have created a nice Github Actions Matrix to reuse as much as possible. I am not going into too much detail here and the latest action setup probably already changed when you read this, but I have linked the specific sections for record. the

1. MacOS is iOS but different—and Google won’t tell you

For the release build of iOS to work, we needed to sign the app. As a matter of fact, the flutter build won’t really work if you don’t have the necessary signatures set up. For iOS nightly we use an Ad-Hoc setup with a few pre-configured internal devices, for the release we used the distribution profiles, both stored as environment secrets as is so commonly used in many tutorials. For MacOS signatures aren’t necessary to build and distribute the App - presumably because MacOS App development pre-dates signed builds. Thus our nightly builds didn’t have any setup for that yet.

Another important difference to note is on the flutter side. While flutters build ipa offer the options --export-options-plist=PATH allowing you to specify certain plist information overrides for that specific build, no such option exists in flutter build macos. Meaning that all the configuration setup inside the macos-folder is and must be used as-is. That is a bit annoying as it means we can’t easily make a local release build without the signatures now but that’s what it is.

One annoying side-effect of Flutter being a lot more popular for building mobile apps is that when you try to Google for information regarding the apple setup needed you’ll almost exclusively find questions and problems for iOS. They then recommend stuff like the export-options-command or other obscure settings you are supposed to change via xcode but that doesn’t actually do anything in the desktop version or doesn’t even exist. Google really doesn’t help you when you get stuck with your Flutter MacOS build.

2. Switching from github environment secrets to git-crypt for signatures and profiles

One thing we wished we had done earlier was switching from storing signatures and provisioning profiles in Github secret environment variables to using git-crypt`. Many tutorials and setups out there recommend using the Github secrets to store, well, secret information like the provisioning profiles and the secrets from the keychain and then have some companion script that puts that into the local Github Action build. That is all good and dandy if you only do that setup once and rarely change it. But I always found it kinda annoying that despite no hint in the Git history a build might fail or pass. Once you go beyond just managing a single profile the scripts are then often falling apart and the increasing number of environment variables becomes very confusing and it is super easy to mess up in converting them into the right base64 because it was soo long ago you did it last.

Rather than storing profiles and the keystore and similar file-based secrets in the secret environment variables we switched to using git-crypt a git extension you can configure that transparently encrypts a subset of files before committing them to the repo. That makes it super easy and simple to update them and still keep the files available. Rather than extracting each secret from the environment into a file we just install git-crypt and have the main password as the action secret that we then use to decrypt the files:

- name: Unlock git-crypt
  if: matrix.with_apple_cert
  run: |
    brew install git-crypt
    echo "$" | base64 --decode > .github/assets/git-crypt-key
    git-crypt unlock .github/assets/git-crypt-key
    echo "Files found:"
    git-crypt status -e

Once we have the files decrypted, we use the commonly used script to import the keychain from the now decrypted file. Technically we wouldn’t even need the extra password for that file, but it also doesn’t hurt:

# Install the Apple certificate and provisioning profile
- name: Install the Apple certificates
  if: matrix.with_apple_cert
  env:
    P12_PASSWORD: ${{ secrets.BUILD_CERTS_P12_PASSWORD }}
    KEYCHAIN_PASSWORD: ${{ secrets.KEYCHAIN_PASSWORD }}
  run: |
    echo "starting in $RUNNER_TEMP"
    # create variables
    CERTIFICATE_PATH=".github/assets/build_certificates.p12"
    KEYCHAIN_PATH="$RUNNER_TEMP/app-signing.keychain-db"
    echo "vars set"
    # import certificate and provisioning profile from secrets
    # create temporary keychain
    echo "creating keychain"
    security create-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"
    echo "setting keychain"
    security set-keychain-settings -lut 21600 "$KEYCHAIN_PATH"
    echo "unlocking keychain"
    security unlock-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"
    # import certificate to keychain
    echo "importing certificate"
    security import "$CERTIFICATE_PATH" -P "$P12_PASSWORD" -A -t cert -f pkcs12 -k "$KEYCHAIN_PATH"
    echo "listing keychains"
    security list-keychain -d user -s "$KEYCHAIN_PATH"

And finally we just take all the now decrypted provisioning_profiles files and copy them where they need to be. All files for all builds in the git repo. Sweet.

- name: Install the Apple provisioning profile
  if: matrix.with_apple_cert
  run: |
    echo "Installing provision profiles"
    mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles/
    cp .github/assets/provision_profiles/* ~/Library/MobileDevice/Provisioning\ Profiles/
    ls -ltas ~/Library/MobileDevice/Provisioning\ Profiles/

Finally don’t forget to clean all that up, regardless of whether the build failed or succeeded!

- name: Clean up keychain and provisioning profile
  if: ${{ always() }}
  run: |
    security delete-keychain $RUNNER_TEMP/app-signing.keychain-db
    rm ~/Library/MobileDevice/Provisioning\ Profiles/*
    rm .github/assets/git-crypt-key

This makes it super easy to update all that data. Got a new provisioning profile? Just put it into that folder. Update to the keystore? Just export the p12-file with the same password again. No base64 conversion, no copying into the Github Secrets - just git commit && push. *chefskiss*.

3. Github Search is the hidden champion

One of Githubs most underrated features is its search. It being the biggest crowd source code knowledge base in the world, including the largest source for all their own configuration files (which the workflow-yamls are one of) their search can be truly amazing. Not everyone who found the hack to make something happen will blog about it or write a stack overflow—this post almost didn’t make it either. But it if works there is a high chance they commit it and it ends up in the Github repo, discoverable via the search.

Similar as Google, Github’s search has many advanced options. For us looking for alternative ways of doing the Flutter build within the Actions, adding the path:.github/workflow flutter macos was the key to unlocking a treasure of knowledge. Mind you that even though code is committed doesn’t necessarily mean it runs, though. But it is how we first found out about the git-crypt idea! And that’s also how we found out about the final upload pattern we ended up using.

Seriously, if you are ever stuck on some Github Action configuration that others probably already attempted try the Github search. Google doesn’t even know about a fraction of it and with the advanced search you can make the queries very specific to your problem.

4. MacOS apps need all their binaries signed

One particularly nasty difference between the iOS and MacOS flutter build is that the latter doesn’t really manage the signing properly for you. Singing the Mac App is different than the iOS one, too: While on iOS you create an ipa-file (effectively a Zip-File) which is then signed as a whole (oversimplified), the “Mac App” is actually a directory with the extension .app. You can’t effectively “sign” directories. Instead the people at Apple decided that what you must do is sign each binary within that app directory and provide these signatures in the directory. This is hidden in the docs somewhere but if you tried to Google for this information, you will only find iOS fixes (see No 1). So I am telling you know.

For most cases that is fairly irrelevant but as we had a bunch of binaries, our own included. We found a script that iterates through the final app and signs each binary with the provided credentials, which we then added to the regular Xcode shell-script build process for release builds. That means that at the end of flutter build macos, we now have an Acter.app directory with all the proper signatures included. Yay.

5. Build code ages quickly

One problem you’ll be facing with the Github Search as well as the Google search answers regardless is that the infrastructure you are building with and against constantly changes. For us, there were several tutorials out there recommending ways of packaging or uploading the app that were outdated to simply not supported anymore for the latest version (this was even worse for building the Windows App). Trying to figure out which is the latest recommended and thus hopefully the longest-lasting code you could write is a tedious and annoying process. Very often you don’t know this isn’t supported anymore until you installed and tried the command. But there is a few tricks to keep in mind, when you find a novel approach you might want to try out: you can check the official docs and see if it is still supported, if it is on Github you can see when it was last run, for StackOverflow and many blogs you can quickly gather whether this is a new or rather old idea. Unfortunately in this space, old often means less likely too still work…

For us the latest—at the time of writing—and recommended way to package and submit the Flutter MacOS app to the Apple Mac Store is:

  1. build the release version of the app with flutter build macos; make sure all binaries are signed (see above)
  2. use productbuild to create a modern .pkg and have it signed: productbuild --component Acter.app /Applications --sign "$APPLE_SIGN_CERTNAME" Acter.pkg
  3. then use altool to upload the .pkg to the Apple Mac AppStore using a private_key credential (which we stored with git-crypt, of course): 
          - name: Upload to App Store
     env:
       APPLE_API_KEY_BASE64: ${{ secrets.APPLE_API_KEY_BASE64 }}
       APPLE_API_KEY_ID: ${{ secrets.APPLE_API_KEY_ID }}
       APPLE_ISSUER_ID: ${{ secrets.APPLE_ISSUER_ID }}
     run: |
       mkdir private_keys
       echo -n "$APPLE_API_KEY_BASE64" | base64 --decode --output "private_keys/AuthKey_$APPLE_API_KEY_ID.p8"
       ls -ltas private_keys
       xcrun altool --upload-app --type macos --file acter-macosx-${{ needs.tags.outputs.tag }}.pkg \
           --bundle-id global.acter.a3 \
           --apiKey "$APPLE_API_KEY_ID" \
           --apiIssuer "$APPLE_ISSUER_ID"
     shell: bash

6. Github artifacts are not a proper package mechanism: measure twice

With that we are all set and everything should work. Yet Apple kept rejecting our app. But only after the upload in the post-processing on the server side, a few hours later we’d receive an email saying something along the lines of:

 ITMS-90238: Invalid Signature - The main app bundle Acter at path acter-macosx-1.23.1116.app has following signing error(s): --prepared:/Volumes/workspace/app_data/SWValidationService/mz_2801087702614282773dir/mz_16205544286065377385dir/global.acter.a3.pkg/Payload/acter-macosx-1.23.1116.app/Contents/Frameworks/Reachability.framework/Versions/Current/. [...]

ITMS-90238: Invalid Signature - The executable at path acter-macosx-1.23.1116.app/Contents/Frameworks/App.framework/App has following signing error(s): bundle format is ambiguous (could be app or framework) . Refer to the Code Signing and Application Sandboxing Guide at http://developer.apple.com/library/mac/#documentation/Security/Conceptual/CodeSigningGuide/AboutCS/AboutCS.html and Technical Note 2206 at https://developer.apple.com/library/mac/technotes/tn2206/_index.html for more information.

ITMS-90238: Invalid Signature - The executable at path acter-macosx-1.23.1116.app/Contents/Frameworks/FMDB.framework/FMDB has following signing error(s): bundle format is ambiguous (could be app or framework) . Refer to the Code Signing and Application Sandboxing Guide at http://developer.apple.com/library/mac/#documentation/Security/Conceptual/CodeSigningGuide/AboutCS/AboutCS.html and Technical Note 2206 at https://developer.apple.com/library/mac/technotes/tn2206/_index.html for more information.

[...] repeated 20 or so times

ITMS-90303: Unable to Sign - This package doesn't meet the current code signing requirements. For more information, see the Code Signing and Application Sandboxing Guide at http://developer.apple.com/library/mac/#documentation/Security/Conceptual/

[...]

TMS-90291: Malformed Framework - The framework bundle App (acter-macosx-1.23.1116.app/Contents/Frameworks/App.framework) must contain a symbolic link 'App' -> 'Versions/Current/App'. Refer to the Anatomy of Framework Bundles for more information.

ITMS-90291: Malformed Framework - The framework bundle App (acter-macosx-1.23.1116.app/Contents/Frameworks/App.framework) must contain a symbolic link 'Resources' -> 'Versions/Current/Resources'. Refer to the Anatomy of Framework Bundles for more information.

ITMS-90292: Malformed Framework - The framework bundle App (acter-macosx-1.23.1116.app/Contents/Frameworks/App.framework) 'Versions' directory must contain a symbolic link 'Current' resolving to a specific version directory. Resolved link target: '${linkTarget}'. Refer to the Anatomy of Framework Bundles for more information.

[...] repeated 20 or so times

The attentive reader might have already noticed the actual problem already. As we had gotten several similar looking emails (especially the top part) from before when the signatures were not properly set up for each binary, we assumed it was something wrong with that part again. Weirdly enough, when doing the entire process manually (rather than via Github Action) it all worked fine and Apple didn’t reject our submission. Weird. So we downloaded the latest Acter.app from the build artifacts to try to see if we could sign and submit that. The download was larger than usual (220mb rather than the usual 140mb we saw for most builds before) but we didn’t really think much about it. Indeed, trying to package and upload this version Apple rejected it again.

So, we look into the insides of the Acter.app build by the Github Action: it is just a folder after all (even though MacOS finder hides it under the right click -> Open Contents). Right away we noticed something odd: all binaries for the frameworks appeared to be in there twice: once under $framework/Versions/Current and once as $framework/Versions/A. That sure explained why it would be about twice the size. Interestingly our nightly builds didn’t show this behavior: there Current was a symlink to A for each as—we’d expect it to be. So although the nightly build system was the baseline we started with, we must have altered something along the way.

Then it hit us: the main difference is that in the nightly job packages the .app-Folder as a tar.bz directly and submits it to the Github release from the build job, while in the publishing action we store the folder as a Github Artifact that a second job after downloads and submits to the store.

Why does that matter? The Github Artefact is stored as a ZIP, too (to save disk space), after all. Yes, but for zipping by default is that symbolic links are resolved. As we are zipping the entire folder the symlinks that is usually Versions/Current -> Versions/A is resolved, meaning the files are stored twice. Yet the signature is only stored once and only for Versions/A (not for Versions/Current). So when we download that zipped version, we have an .app-folder with each framework version stored twice yet only a signature for one (and the file having about twice the size). Looking at the error messages sent by Apple, the last batch of errors even gives a hint to that problem.

Finding that issue, one and off, took us a month. Yet the fix was small and trivial: we moved the productbuild to create a .pkg-file from the publishing job into the build-job and store that .pkg-file as the artefact. Problem solved.

These are just a few things we wished we had known before. Do you have any additional tips for that—apparently niche—Flutter MacOS build systems you wished someone had told you before? Let me know!