HTTP (hypertext transfer protocol) is a communication protocol that transfers data between client and server. HTTP requests are very essential to access resources from the same or remote server. In Golang, the net/http package comes with the default settings that we need to adjust according to our high-performance requirement.
For setting up HTTP clients for making requests, most programming languages have different frameworks in place. We will take a hands-on approach in the coming sections to explore how HTTP requests can be made in Golang or Go, as I will refer to the language for the rest of the post.
\nWhile working on the Golang projects, I realized that improper configuration of HTTP might crash your server anytime.
\nIn the time when I was working with HTTP Client, I Observed some problems and their solutions, listed below:
\nThe HTTP client does not contain the request timeout setting by default.\nIf you are using http.Get(URL) or &Client{} that uses the http.DefaultClient. DefaultClient has not timeout setting; it comes with no timeout
Suppose the Rest API where you are making the request is broken, not sending the response back that keeps the connection open. More requests came, and open connection count will increase, Increasing server resources utilization, resulting in crashing your server when resource limits are reached.
\nvar httpClient = &http.Client{\n Timeout: time.Second * 10,\n}For the Rest API, it is recommended that timeout should not more than 10 seconds.\nIf the Requested resource is not responded to in 10 seconds, the HTTP connection will be canceled with net/http: request canceled (Client.Timeout exceeded ...) error.
\nBy default, the Golang Http client performs the connection pooling. When the request completes, that connection remains open until the idle connection timeout (default is 90 seconds). If another request came, that uses the same established connection instead of creating a new connection, after the idle connection time, the connection will return to the pool.
\nUsing the connection pooling will keep less connection open and more requests will be served with minimal server resources.
\nWhen we not defined transport in the http.Client, it uses the default transport Go HTTP Transport
\nDefault configuration of the HTTP Transport,
\nvar DefaultTransport RoundTripper = &Transport{\n\t...\n\tMaxIdleConns: 100,\n\tIdleConnTimeout: 90 * time.Second,\n\t...\n}\n\nconst DefaultMaxIdleConnsPerHost = 2MaxIdleConns is the connection pool size, and this is the maximum connection that can be open; its default value is 100 connections.
\nThere is problem with the default setting DefaultMaxIdleConnsPerHost with value of 2 connection,\nDefaultMaxIdleConnsPerHost is the number of connection can be allowed to open per host basic.\nMeans for any particular host out of 100 connection from the connection pool only two connection will be allocated to that host.
With the more request came, it will process only two requests; other requests will wait for the connection to communicate with the host server and go in the TIME_WAIT state. As more request came, increase the connection to the TIME_WAIT state and increase the server resource utilization; at the limit, the server will crash.
t := http.DefaultTransport.(*http.Transport).Clone()\nt.MaxIdleConns = 100\nt.MaxConnsPerHost = 100\nt.MaxIdleConnsPerHost = 100\n\t\nhttpClient = &http.Client{\n Timeout: 10 * time.Second,\n Transport: t,\n}By increasing connection per host and the total number of idle connection, this will increase the performance and serve more request with minimal server resources.
\nConnection pool size and connection per host count can be increased as per server resources and requirements.
\nIn this article, we discussed the problems around the 'net/http' client default configurations. By changing some of the default settings of HTTP Client, we can achieve a High-performance HTTP client for production use. If you want to learn more about http, here is an interesting post on HTTP security headers If you like what you read, share your thoughts in the comment section.
\n","frontmatter":{"date":"January 12, 2021","updated_date":null,"title":"How to Use the HTTP Client in GO To Enhance Performance","tags":["Golang","HTTP","Performance"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.7699115044247788,"src":"/static/68276d4b4277ad09e329598038e91917/14b42/index.jpg","srcSet":"/static/68276d4b4277ad09e329598038e91917/f836f/index.jpg 200w,\n/static/68276d4b4277ad09e329598038e91917/2244e/index.jpg 400w,\n/static/68276d4b4277ad09e329598038e91917/14b42/index.jpg 800w,\n/static/68276d4b4277ad09e329598038e91917/47498/index.jpg 1200w,\n/static/68276d4b4277ad09e329598038e91917/ec6c5/index.jpg 1280w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Mayank Agarwal","github":"mayankagrwal","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/what-is-the-new-go-1-15/"},"html":"Go announced Go 1.15 version on 11 Aug 2020. Highlighted updates and features include Substantial improvements to the Go linker, Improved allocation for small objects at high core counts, X.509 CommonName deprecation, GOPROXY supports skipping proxies that return errors, New embedded tzdata package, Several Core Library improvements and more.
\nAs Go promise for maintaining backward compatibility. After upgrading to the latest Go 1.15 version, almost all existing Golang applications or programs continue to compile and run as older Golang version.
\nDownload the latest Go Version from Golang.
\nGo 1.14 version announced some changes for the macOS. According to announcement
\n\n\nGo 1.14 is the last release that will run on macOS 10.11 El Capitan. Go 1.15 will require macOS 10.12 Sierra or later.
\n
The lastest macOS 10.15 (Catalina) is no longer supporting 32-bit binaries. Go 1.14 will likely to be the last Go release to support 32-bit binaries on iOS, iPadOS, watchOS, and tvOS (the darwin/arm port). Go continues to support the 64-bit darwin/amd64 port.
\nGo now generates Windows ASLR executables when -buildmode=pie cmd/link flag is provided. Go command uses -buildmode=pie by default on Windows.
The -race and -msan flags now always enable -d=checkptr, which checks uses of unsafe. Pointer. This was previously the case on all OSes except Windows.
Mobile applications were crashing in some devices due to some invalid lld linker versions so now avoiding the crashing the latest Go 1.15 version explicitly selects the lld linker available in recent versions of the NDK. This explicitly lld linker avoids crashes on some devices. it becomes the default NDK linker in a future NDK version.
\nAlways new Go 1.15 brings many minor/major performance improvements. The first major improvement has reduced the executable's binary size by almost 5% - 10% as compared to Go 1.14. As Brad Fitzpatrick tweet, test program down from 8.2MB in Go 1.14 to 3.9MB in 1.15.
\nGo 1.15 has improved the code build process now unused code is eliminating in the new linker, along with several targeted improvements, such as Clements's CL 230544, which reduces the number of stack and register maps included in the executable.
\nAlways in every new version release Go come with various minor changes and updates to the library. I m including some useful and important changes
\nA new embedded tzdata package was added that permits embedding the timezone database into a program. Importing this package (as import _ \"time/tzdata\") which allows the program to find timezone information without the timezone database on the local system. We can also embed the timezone database by building with -tags timet/zdata. This approach increases the size of the program by about 800 KB. This might be useful and can test some code with the virtualized environments like Go playground.
\nGo older versions were using CommonName field on X.509 certificates as a hostname if there is no Subject\nAlternative Names, would be disabled by default. If Still want to use this legacy behavior then we need to add x509ignoreCN=0 in the GODEBUG environment variable.
\nThe net/url package adds a new URL. The redacted () method that returns the URL as a string. This is proposed in the Issue 34855. It is a very useful improvement for audit logging and security. It's a simple derivation from the URL.String() that masks the password if exists from the string being passed. It does not modify at all to the URL itself but a copy of it.
Go 1.15 has released with various improvements, bug fixes. We can get all improvements, closed proposals and features details for the Go GitHub repository.
\nMigrating your existing code to a new programming language is a very tedious task. We need to have a proper set of requirements and some obvious benchmarking between the new and existing technology.
\nSo here our journey started to transform (not convert) our existing code to a new language. You might be thinking why I have used the word transformation instead of conversion, so -
\nConversion: Conversion is just about writing your existing code in different languages as it is.
\nTransformation: Transformation is about leveraging all possible benefits of a new language and convert accordingly.
\nThe idea of migrating code to a new programming language started with a simple requirement: we wanted to take our cloud solution to on-premise as well. Till now, our services were written in NodeJS for a cloud solution and it was doing pretty well. The performance was also good but there is nothing that cannot be improved upon. So, along with performance improvement, we started to find out that can we make a GUI or CLI using existing NodeJS code that can take our solution on-premise?
\nAnswer for that was practically YES. but, technically NO. If we’re writing apps with a GUI, Node can’t do it on its own. We had to use some other projects which provide GUI creation capabilities (eg: ElectronJS, NW.js, etc). Making CLI seemed doable but there were also problems like the inablility to have single file distribution for our code. And the ability to download a single file and execute commands - without an installer, or even a setup process - does wonders for a user. This also makes the product easily backward-compatible.
\nAlso, another factor came in that it was better to switch to a technology that works closely with gRPC.
\nSo, we drew out a few goals for our new solution
\nAfter research with our requirement, we were able to figure out that we can achieve these with GoLang, developed by Google.\nOn the basis of our research, we also did some benchmarking on different node js frameworks and GoLang.
\nBenchmarking sever we’ve used with below configurations
\nThe benchmarking result was pretty much similor to what we are thinking. Here are the benchmarking results -
\nThe below chart is showing that the Number of requests is very high compared to other node js frameworks.
\nAll the data clearly shows the direction so we decided to move to GoLang.
\n","frontmatter":{"date":"August 11, 2020","updated_date":null,"title":"A journey from Node to GoLang","tags":["NodeJs","Golang","Performance"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/0cb80044e5634c85f2b1002c7c3661d8/2a4de/node-go.png","srcSet":"/static/0cb80044e5634c85f2b1002c7c3661d8/69585/node-go.png 200w,\n/static/0cb80044e5634c85f2b1002c7c3661d8/497c6/node-go.png 400w,\n/static/0cb80044e5634c85f2b1002c7c3661d8/2a4de/node-go.png 600w","sizes":"(max-width: 600px) 100vw, 600px"}}},"author":{"id":"Narendra Pareek","github":"pareek-narendra","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/custom-encoders-in-the-mongo-go-driver/"},"html":"The official supported drivers for Mongo for Go have been officially released for several months now and is slowly seeing more usage. Although many of the features of standard Mongo drivers have been implemented within the supported driver, the documentation for a majority of the features are still in the process of being created. This post will provide some information on how to set up custom JSON encoders so you can change how BSON operators are processed whenever Mongo documents are marshalled into JSON. For this blog, a custom date and object id encoder will be created to handle the BSON operators and transform them into more easily processed JSON.
\nThis blog assumes that you have a basic understanding of Go and how to write some code in the language. If you want to follow along with the guide, you will also require a working Mongo instance, as well as your environment set up for Go.
\nBefore interacting with the database, we can set up a quick Go application that finds data from a Mongo database and outputs it. For this example, the Mongo database is set up with a collection containing documents with a name, a date and the object id. For our blog, the Mongo document that is being worked with is quite simple:
\n{\n "_id" : ObjectId("5de6b0850f3c7b5cce642529"),\n "Name" : "dateOne",\n "Date" : ISODate("2019-04-08T11:55:02.658Z")\n}A simple main function that retrieves data from Mongo and outputs the JSON string is as follows:
\nfunc main() {\n\tvar result struct {\n\t\tID primitive.ObjectID `bson:"_id"`\n\t\tName string `bson:"Name"`\n\t\tDate primitive.DateTime `bson:"Date"`\n\t}\n\n\tctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)\n\tclient, connErr := mongo.Connect(ctx, options.Client().ApplyURI(""))\n\n\tif connErr != nil {\n\t\tlog.Fatal(connErr)\n\t}\n\n\tcollection := client.Database("test").Collection("dates")\n\n\tfilter := bson.M{"Name": "dateOne"}\n\tfindErr := collection.FindOne(ctx, filter).Decode(&result)\n\tif findErr != nil {\n\t\tlog.Fatal(findErr)\n\t}\n\n\tfmt.Print(result.Date)\n\n\tdata, writeErr := bson.MarshalExtJSON(result, false, false)\n\tif writeErr != nil {\n\t\tlog.Fatal(writeErr)\n\t}\n\n\tfmt.Print(string(data))\n\tcancel()\n}The function MarshalExtJSON is a part of the bson package that is a part of the driver. It uses a default registry to define the rules when converting Mongo primitive types into JSON. The arguments it takes are the Go struct being passed in, whether the result should be returned in canonical form (For more details, check this link, and whether HTML strings are escaped.
\nThe output of this function when reading our document produces a JSON string with all the BSON operators included.
\n{\n\t"_id": {\n\t\t"$oid":"5de6b0850f3c7b5cce642529"\n\t},\n\t"Name":"dateOne",\n\t"Date":{\n\t\t"$date":"2019-04-08T11:55:02.658Z"\n\t}\n}Using custom encoders, we can change how the bson package marshals the JSON and remove all the BSON operators. The first thing to do is to define package level variables that will hold the types of the Mongo primitives that need to be changed in the JSON response. Outside the main function, include these two lines:
\nvar tOID = reflect.TypeOf(primitive.ObjectID{})\nvar tDateTime = reflect.TypeOf(primitive.DateTime(0))Next, we can make new functions to customize how these types are handled. For the date encoding, we want to have the Date key matched with the returned date string.
\nfunc dateTimeEncodeValue(ec bsoncodec.EncodeContext, vw bsonrw.ValueWriter, val reflect.Value) error {\n\tconst jDateFormat = "2006-01-02T15:04:05.999Z"\n\tif !val.IsValid() || val.Type() != tDateTime {\n\t\treturn bsoncodec.ValueEncoderError{Name: "DateTimeEncodeValue", Types: []reflect.Type{tDateTime}, Received: val}\n\t}\n\n\tints := val.Int()\n\tt := time.Unix(0, ints*1000000).UTC()\n\n\treturn vw.WriteString(t.Format(jdateFormat))\n}The parameters in this function are pre-defined within the driver package of the driver. This involves passing in the BSON encoding context, the value writer and the value that is being processed. The value writer and the value are the parameters that we use within our function.
\nFirst, we check if the value passed in is the correct type. If there is an issue with the value, then an error is thrown during the encoding. Next, we can set the format we want our date string to appear in, and process our value into a Go Time object. After that, we can use the value writer to write the formatted date into our JSON string.
\nSimilar to the custom date encoding, the object id encoding function has the same structure.
\nfunc objectIDEncodeValue(ec bsoncodec.EncodeContext, vw bsonrw.ValueWriter, val reflect.Value) error {\n\tif !val.IsValid() || val.Type() != tOID {\n\t\treturn bsoncodec.ValueEncoderError{Name: "ObjectIDEncodeValue", Types: []reflect.Type{tOID}, Received: val}\n\t}\n\ts := val.Interface().(primitive.ObjectID).Hex()\n\treturn vw.WriteString(s)\n}First, we check if the type is valid, and if it is, we simply write out the object id as a hex string. To create the hex string, we first convert the value into an interface, then cast it as an object id to allow the Hex function to be called.
\nNow that our two functions are created to handle the BSON types, we can create our custom registry.
\nfunc createCustomRegistry() *bsoncodec.RegistryBuilder {\n\tvar primitiveCodecs bson.PrimitiveCodecs\n\trb := bsoncodec.NewRegistryBuilder()\n bsoncodec.DefaultValueEncoders{}.RegisterDefaultEncoders(rb)\n bsoncodec.DefaultValueDecoders{}.RegisterDefaultDecoders(rb)\n\trb.RegisterEncoder(tDateTime, bsoncodec.ValueEncoderFunc(dateTimeEncodeValue))\n\trb.RegisterEncoder(tOID, bsoncodec.ValueEncoderFunc(objectIDEncodeValue))\n\tprimitiveCodecs.RegisterPrimitiveCodecs(rb)\n\treturn rb\n}In this function, we are setting up our registry to use our custom encoders, while also setting up default encoders for all other types that are being marshalled. Using the bsoncodec package, the NewRegistryBuilder function is called to initialize a new registry. Next, we set up this registry with the default encoders and decoders that are pre-programmed in the driver using RegisterDefaultEncoders. During this step, we can also register encoders and decoders for raw types by calling RegisterPrimitiveCodecs. Finally, we can include our custom encoders by calling RegisterEncoder on the custom registry struct. Our encoder will be called whenever the type defined in the first argument in RegisterEncoder is found in the struct being marshalled.
\nTo ensure our new custom registry is being used, we have to do two things. The first is to build the custom registry. This should be done before the marshalling of the struct.
\nvar customRegistry = createCustomRegistry().Build()Next, we need to change the marshalling method to use the new custom registry instead of the default one. We can do this by changing the method being used to marshal the struct.
\ncustomData, writeErr := bson.MarshalExtJSONWithRegistry(customRegistry, result, false, false)The function MarshalExtJSONWithRegistry takes the same parameters as MarshalExtJSON, but also requires a custom registry to be passed in as the first parameter. We pass in the custom registry we created into the method to allow the marshalling to use our custom encoders. As a result, when printing our new JSON string output, we get this:
\n{"_id":"5de6b0850f3c7b5cce642529","Name":"dateOne","Date":"2019-04-08T11:55:02.658Z"}The Mongo driver for Go is still in active development and new versions continue to be released. Hopefully this blog provides some useful information on how the driver works while new documentation is being created.
\n","frontmatter":{"date":"December 03, 2019","updated_date":null,"title":"Custom Encoders in the Mongo Go Driver","tags":["Go","Golang","MongoDriver"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.25,"src":"/static/0bf88d6b53aaaf119a2556b41f4c8383/ee604/mongo-drivers.png","srcSet":"/static/0bf88d6b53aaaf119a2556b41f4c8383/69585/mongo-drivers.png 200w,\n/static/0bf88d6b53aaaf119a2556b41f4c8383/497c6/mongo-drivers.png 400w,\n/static/0bf88d6b53aaaf119a2556b41f4c8383/ee604/mongo-drivers.png 800w,\n/static/0bf88d6b53aaaf119a2556b41f4c8383/30c15/mongo-drivers.png 936w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Chris Yee","github":null,"avatar":null}}}}]}},"pageContext":{"tag":"Golang"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}