One non-obvious part about git-annex is "location tracking" system [0], which "keeps track of in which repositories it last saw a file's content."
The result of this is that any "checkout" operations will, by default, modify source repo, and then this information will be eventually propagated to all the clients. Before you delete the repository, you'll want to de-register any git-annex files to let others know they are no longer available. This is pretty big departure from standard git approach where you can delete any checkouts without any repercussions.
We've tried to use git annex for large file storage in a small team, and our metadata kept being "gummed up" with all the repos which no longer exists -- temporary CI checkouts, servers which have been formatted, old user laptops, etc.. So we dropped it in favor of "git lfs" and then a custom system.
When I started researching version control tools for large files, I remember feeling like git-annex and Git LFS were awkwardly bolted onto Git; Git simply wasn't designed for large files. Then I found DVC[1], and its approach rang true for me. However, after using DVC for a year or so, I grew tired of its many puzzling behaviors (most of which are outlined in the README at [0]). In the end, I built the tool I wanted for the job -- one that is exceptionally simple and fast.
[0]: https://github.com/kevin-hanselman/dud
[1]: https://dvc.org/