article archive

Go compiler optimizations for structs

This article looks into the optimizations of the golang compiler with regard to different usage of structs. It answers the following questions:

  • Does it avoid unnecessary structure copies?
  • Does it inline where necessary?

To answer theses questions, I wrote a small example and decompiled it to see how well it behaves.
For all the examples, I used the compiler version 1.15 and a main package with a main function calling all the others. I always tried with both pointer and value operations.

Small data structure

First, I defined a small data structure:

type A struct{ contained int }

Then, I added some receivers:

func (a *A) mutate() { a.contained += 123 }
func (a A) doNotMutate() { a.contained += 123 }

With the pointer receiver, the decompilation shows that it’s a simple load & add & store.
For the value receiver on the other hand, as the copied structure will drop on return, it can be a NOP, and it is indeed the case.
Let’s see what happens when a function calls doNotMutate:

func doNotMutateAWithValue(a A) { a.doNotMutate() }
func doNotMutateAWithPtr(a *A) { a.doNotMutate() }

Here, the first is also a NOP, as expected.
The second one simply checks for a nil pointer, so as expected. At call sites, it isn’t called at all, perfect!
Then I tried calling mutate from a function:

func mutateAWithValue(a A) { a.mutate() }
func mutateAWithPtr(a *A) { a.mutate() }

When mutating a discarded value, it actually performs the unneeded computation, too bad. The call is inlined though.
The second one is inlined and acts as expected.
But nicely enough, at call sites, mutateAWithValue is not called at all, that’s a relief.

So let’s recap: for a small struct, the code generated for the methods might not be as efficient as expected, but at call sites, it’s optimized away.

Big data structure

Then I tried with a bigger struct to see if that’s as efficient.

type B struct{ a, b, c, d, e, f, g, h, i, j uint64 }

func (b *B) mutate() { b.i += 123 }
func (b B) doNotMutate() { b.i += 123 }

The first call is correct and simple, as for A.mutate.
But the second call performs the computation on a discarded value. Luckily this is optimized away when calling B.doNotMutate from a function.

Calling the receivers from a function that takes the structure as an argument shows some missing optimizations:

func doNotMutateBWithValue(b B) { b.doNotMutate() }
func doNotMutateBWithPtr(b *B) { b.doNotMutate() }

And it all goes crumbling: in both calls, it actually copies the whole structure before executing the useless computation and discarding the value.
The call sites aren’t saving it: the functions using a value and a pointer argument both copy the structure before discarding it.

func mutateBWithValue(b B) { b.mutate() }
func mutateBWithPtr(b *B) { b.mutate() }

Similarly to the A version, the compiler creates a computation even when not needed, but inlined. When calling B.mutate  in mutateBWithValue it is much worse: the generated code copies the structure before dropping it.

Summary

After all these experiments, I can say that the go 1.15 compiler optimizes structure receivers depending on the size of the structure:

  • small structs are well handled, inlined and eluded as efficiently as possible
  • bigger structs perform badly, as they are copied when passing through a function
  • inlining is done pretty intensely, as no call is created for any of the functions or methods
  • uses of value & pointer receivers are correctly optimized

Failing OmniLedger login

Connection Failures

There are some common reasons why the OmniLedger login can fail. So far we have seen the following ones:

  • Corporate firewalls
  • Private browser settings

If you have a corporate firewall that blocks access to non-standard ports, the only thing you can do is to remove the VPN. This is often enough to let OmniLedger login work, at least if you’re not in your office.

Concerning the private browser settings, the problem is the following: OmniLedger login doesn’t rely on passwords, but rather it relies on a secret private key. This private key is stored in your browser. Unfortunately storing data is often abused, so some browsers are set to private navigation all the times. This means that the browser will not store the private key. So the OmniLedger login cannot work! Here are some settings how you can overcome that:

  • Firefox
    • Use a normal window instead of a “Private window”
    • In the preferences of Firefox, under “Privacy and Security”, the “History” setting should be on “Remember History”
  • Chrome
    • Use a normal window instead of an “Incognito window”
    • In the settings of Chrome, under “Privacy and Security”, make sure that
      • “Block all cookies” is NOT selected
      • “Clear cookies and site data when you quit chrome” is INACTIVE
  • Safari
    • Use a normal window instead of a “Private window”
    • In the preferences of Safari, under “Privacy”, make sure that “Block all cookies” is DESELECTED
  • iOS
    • In the Settings, under “Safari”, remove “Block all Cookies”

Maturity Evaluations

Maturity Evaluation

The C4DT/Factory works to bring EPFL’s great ideas to our partners. We do so by working on the existing code created in the labs. Because most of the code written in the labs has the goal to create graphs in a paper, it is not directly usable in another project. For this reason we created the Maturity Evaluation. In short, this is a simple Google Spreadsheet. with different evaluations to fill out. At the end we see how the code can be improved. The Maturity Evaluation also gives us an indication of the overall quality of the code: Prototype, Intermediate, or Mature.

The showcase.c4dt.org includes the result of the Maturity Evaluations we did so far. Every lab that is associated. with C4DT can ask us to evaluate their code. Then we will add it to the showcase.

The Maturity Evaluation has the following sections:

  • Code – which evaluates the readability and documentation of the code
  • Culture – everything around the code, including setup of the github account, pull requests, issues, and many more
  • Context – paper references, integration with other code

You can get a copy of the spreadsheet here: Maturity Evaluation V0.5 There is also a document explaining the different evaluations and what they refer to: Maturity Evaluation Explainer V0.5

If you have questions or comments, please don’t hesitate to get in contact with me: linus.gasser@epfl.ch

TLDR for “Managing Technical Quality”

Some notes from https://lethain.com/managing-technical-quality/ – most of the comments are copy/pasted, so ‘I’ is the author… In italic some more C4DT-specific notes.

  • Technical quality is a long-term game. There’s no such thing as winning, only learning and earning the chance to keep playing.
  • Do the quick stuff first!

Hot Spots

  • Before defining new processes, measure the problem at hand, identify where the bulk of the issue occurs, and focus on precisely that area.

Maturity Evaluation application, work on red boxes of column I

Best Practices

When you’re rolling out a new practice, remember that a good process is evolved rather than mandated. The handful that I’ve found most helpful to adopt early are version control, trunk-based development, CI/CD, and production observability (including developers on-call for the systems they write), and working in small, atomic changes.

Maturity Evaluation, yellow boxes, needs buy-in from the lab and regular follow-ups.

Leverage Points

  • As you look at how software changes over time, there are a small handful of places where extra investment preserves quality over time, both by preventing gross quality failures and reducing the cost of future quality investments.
  • I call those quality leverage points, and the three most impactful points are interfaces, stateful systems, and data models.

This is probably the last point that we can influence as the C4DT Factory. This requires that the lab is OK with us working on the software artefact in question.

Technical Vectors

One sure-fire solution to align technical direction is to route all related decisions to the same person with Architect somewhere in their title. This works well, but is challenging to scale.

More decentralized approaches include:

  • Give direct feedback.
  • Articulate your visions and strategies
  • Encapsulate your approach in your workflows and tooling
  • Train new team members during their onboarding

Measure Technical Quality

My experience is that it is possible to usefully measure codebase quality, and it comes down to developing an extremely precise definition of quality.

After you’ve developed the definition, this is an area where instrumentation can be genuinely challenging, and instrumentation is a requirement for useful metrics.

Technical Quality Team

When spinning up and operating one of these teams, some fundamentals of success are:

  • Trust metrics over intuition
  • Rotate with developers

Quality Program

Operating organizational programs is a broad topic about which much has been written

  • Identify a program sponsor
  • Identify program goals for every impacted team and a clear path for them to accomplish those goals
  • Build the tools and documentation to support teams towards their goals

Recover data on ByzCoin

How to get data back on ByzCoin

OK, now it happened – I lost my addressbook on ByzCoin. As we’re using the Partner-login internally, I have a list of all accounts I created on the blockchain. Currently it’s not encrypted using Calypso yet, work in progress. So my account on Byzcoin has a slice of all IDs of all accounts I created. This is useful when a user loses his account and wants it recovered. As long as the user did not remove his trust in me, I can do that using a special recovery-rule in DARCs.

Problem is that I deleted my addressbook 🙁 It happened when I was working on reviving an old mobile app for use on the blockchain: Proof of Personhood. The nice thing is that you can link the account on the blockchain with the app. The bad thing is that the app deletes the addressbook! Well, fixed that in the meantime. But, too late.

But ByzCoin is a blockchain, so the information is still stored somewhere in the blocks, so let’s recover it.

The ByzCoin Explorer

I wanted to use https://stackblitz.io/github/c4dt/ol-explorer, but for some bitrotting this does not work anymore. The CEO of stackblitz was quite surprised, but, well, it still doesn’t work. So installing it locally and working locally with it:

git clone https://github.com/c4dt/ol-explorer
cd ol-explorer
npm ci
npm start

This is some hacky way to interact with ByzCoin: all the needed libraries are loaded, the test-network of DEDIS is preconfigured, and you can connect to your account on byzcoin. For programming, you can use your preferred IDE with all the automatic completion and documentation that is available. Would be very nice to be able to use stackblitz, so you would not even need to install anything. But welll…

OK, first thing is to connect to your user on byzcoin: supposing you have an account, on the omniledger-demo, chose Add Device and enter some name. Once the device has been added to your account, copy the link, and on localhost:4200 (if your npm start is still running) click on Attach User and copy the url from the new device.

Getting old Information

Now that the blockchain explorer is linked to our account, we have write-access to it on ByzCoin, so we need to be careful. If we bust it, there might be no way back at all… In the ngOnInit method in src/app/app.component.ts, add the following after the this.logPP…. line.

const versions = await this.bcs.bc.getAllInstanceVersions(this.bcs.user.credStructBS.id);
for (const version of versions){
    this.log(`Version ${version.statechange.version} at block ${version.blockindex} with length ${version.statechange.value.length}`);
}

This will ask one of the nodes to return all known instance-versions and print them out. Unfortunately, this is not a reliable method. The reason is that the nodes do this on a best-effort basis. They don’t need to hold the whole blockchain, as currently about 25 blocks out of the 100’000 available are enough to go from the genesis-block to the latest block. So they always hold the latest version of an instance, but not necessarily all versions. So it might take some F5 reloadings to get old data from a random node…

As it prints the version-number and the length of the data, I simply looked at the place where the length of the data all of a sudden decreased – this is where my addressbook got deleted!

17:06:48 -> 
Version 778 at block 100218 with length 2961

17:06:48 -> 
Version 777 at block 100214 with length 2959

17:06:48 -> 
Version 776 at block 100213 with length 4913

17:06:48 -> 
Version 775 at block 100211 with length 4913

So the last good version of my credential instance seems to be version 776.

Setting new data

This part is not for the faint of the heart. It overwrites the data on the blockchain, so if you remove for example all access delegations, nobody will be able to recover anything from your account. Which is very nice if you want to delete it, not so nice if you want to fix things.

Before I overwrote the data, I actually had a look at it if it made sense. The following code gets version 776 from my instance, the one with the bigger data, and prints out the addressbook. The addressbook is stored in the Attribute of the credential under 1-public/contactsBuf as an array of bytes:

// Get the old version of the credential instance
const contactInstance = await this.bcs.bc.getInstanceVersion(this.bcs.user.credStructBS.id, Long.fromNumber(776));
const credStruct = CredentialStruct.decode(contactInstance.statechange.value);
// Extract the addressbook and display it
const contactsBuf = credStruct.getAttribute("1-public", "contactsBuf");
this.log(contactsBuf);

That looked good – it was a long stream of bytes, which I hoped was my addressbook. To play it safe, I could’ve used the OL-demo under Contacts and copy 64 hex-bytes at a time using Link Contact. But there were many contacts, so I double-checked the following and ran it:

this.bcs.user.executeTransactions((tx) => {
    this.bcs.user.credStructBS.credPublic.contacts.setInstanceSet(tx, new InstanceSet(contactsBuf));
})

This creates a new transaction and executes it. The dynacred-library allows for updating the structures by passing this transaction tx variable to specific methods. Chaining multiple instructions is done by calling multiple methods in the callback of the executeTransactions method. The transaction will be signed and sent to ByzCoin, and if all instructions are valid, the global state will be updated with the new data:

MY ADDRESSBOOK IS BACK!

Recovering an account

So, in fact this was great, as somebody just asked me to recover his account. With my addressbook back, I could search his account, and, as he still trusts me, click on Recover. This is similar to adding a new device to the account, which creates an URL that can be used to link a browser to the account. So I sent this URL over email. Once clicked on the URL, it will connect to demo.c4dt.org/omniledger and store the private key in the browser. Then the person should be ready to use it. Or not, if there is a corporate firewall 🙁