Kafka-Ops: yet another CLI utility to automate Kafka cluster management

Date August 12th, 2020 Author Vitaly Agapov

What we think we’ll do is often not what we end up doing. It isn’t always in our own control, our life.

Guy Gavriel Kay «A Brightness Long Ago»

kafka

This is just an introduction of my CLI tool for idempotent management of Kafka cluster objects (topics and ACLs for now). It is already adopted by our CI/CD pipelines for preparing dedicated or multi-tenant Kafka clusters during deployment of the microservices that act as producers. So we've got rid of any manual management of Kafka topics and ACLs.

The utility is called Kafka-Ops and its source code and releases are available at https://github.com/agapoff/kafka-ops

All the needed documentation is provided in README so I won't post it here. Please refer to Github, use, feedback, contribute.

Tags: ,
Категория: Linux | Нет комментариев »


Nagios style plugin for Kubernetes monitoring

Date July 4th, 2018 Author Vitaly Agapov

“Last man standing," Amos replied with another grin. "It's in my job description.”

James S.A. Corey «Cibola Burn»

kubernetes-icinga

Usually when one thinks about the active monitoring of Kubernetes health then the Prometheus + Alertmanager comes to his mind. This is quite obvious way especially when Prometheus is already an almost mandatory component of the Kubernetes cluster. But what if I am happy with my Icinga/Zabbix/"even Nagios" setup which is monitoring my beyond-Kubernetes stuff and I want to keep everything in one place, manage the contacts, notifications as I used to? Then I need a plugin. And I have one. It is right here:

https://github.com/agapoff/check_kubernetes

The repo has a detailed README so I won't repeat it here.  Just use it, feedback and contribute.

Tags: ,
Категория: Kubernetes | Нет комментариев »


Yet another Fluentd deployment for Kubernetes

Date June 27th, 2018 Author Vitaly Agapov

Accept the pain, but don't accept that you deserved it.

Brandon Sanderson «Oathbringer»

fluentd-k8s-esIt is always mandatory to make logs from the services available for developers and other involved persons. And this access is expected to be not too complicated, without kubectl magic and dashboards (when we are talking about Kubernetes). It is quite easy to start exporting all logs from the Kubernetes cluster to Elasticsearch. The most straightforward way is:

 

Читать далее »

Tags: , , ,
Категория: Kubernetes | Нет комментариев »


Yet another Prometheus deployment for Kubernetes

Date June 22nd, 2018 Author Vitaly Agapov

— There's a right thing to do.
— You don't have a right thing, friend. You've got a whole plateful of maybe a little less wrong.

James S.A. Corey «Leviathan Wakes»

grafana1

I have kept silence here for some time. But at last I have something short to say. Recently I was trying to prepare gathering of metrics from bare-metal Kubernetes clusters for observing them from my favourite Grafana. And it appeared that it's quite difficult to do that using the official documentation and blog posts. Moreover the presented Grafana dashboards were not good enough for me. So when I finished my investigations and experiments and when I was finally happy with my dashboard visualizations I pushed them to Github. The repository can be found here: https://github.com/agapoff/kubernetes-prometheus. It contains the K8s manifests for deploying Prometheus with node-exporter and kube-state-metrics apps and the appropriate ConfigMap. It also contains the json files for Grafana dashboards which can be imported and used out of the box (you need only to create the Prometheus datasource).

Feel free to feedback and contribute.

Tags: , , ,
Категория: Kubernetes | Нет комментариев »


Nagios style plugin for Solace Router monitoring

Date March 21st, 2017 Author Vitaly Agapov

Honor is dead. But I'll see what I can do.

Brandon Sanderson «Words of Radiance»

solace monitoringSolace documentation contains information on some Nagios integration options and a mythical plugin for monitoring Solace from Nagios. But this plugin can not be obtained from Solace site and even from Solace support. The exact quotation from Solace support sounds as "our Nagios plugins are not a product, they are projectware that we provide as part of a paid services engagement related to integration with Nagios". Quite uncomfortable, huh.

So I have prepared own Nagios style plugin (it can obviously be used in Icinga, Shinken etc.) which covers all our needs. It makes use of SEMPv1 management protocol. This protocol is described as legacy in the official documentation but SEMPv2 is not ready yet for usage.

 

Читать далее »

Tags: ,
Категория: Linux, Perl | Нет комментариев »


How to monitor Linux network quality for different directions

Date December 7th, 2016 Author Vitaly Agapov

The legends have two problems. They are not believed in or they are believed in too much.

Alexey Pehov «The Sentinel»

tcp-retrans

Sometimes we might need to know if the network connection between our host and other peers is good enough. The main quality attributes of the network connection are badwidth (quite easy to explore) and the packet loss rate (quite difficult to explore). Of course we can monitor the overall TCP retransmit rate (for example using something like netstat -s | grep -i retrans), build the graph and stare at it. But we'll never know which direction is loosing the packets and where is the problem itself.

I have not found the ready-to-use solution and created my own. You can see the results right here on the screenshot.

 

Читать далее »

Tags: ,
Категория: Linux, Perl | Нет комментариев »


Generate RPM SPEC changelogs in Git

Date November 8th, 2016 Author Vitaly Agapov

You have to be a bit of a liar to tell a story the right way.

Patrick Rothfuss «The Name of the Wind»

rpm-gitThis little note will be useful for those who is lazy enough for dealing with changelogs in RPM spec-files and lazy enough to implement own solution. I will present the script to work from inside the git repository. It can check if the changelog complies the current version and update the changelog accordingly if needed. No git tags are needed.

There are two variants of the script. One is for running on the build server (Jenkins/Bamboo/TeamCity…) right before building the RPM. And the second one is the post-commit hook. Second option can be useful if your build server has no write access to the repository or if you'd like to keep git history cleaner (this option doesn't create extra commits).

The script needs rpmspec utility to be installed. So if you are using some non-RH distro just look for the needed package. In Ubuntu you need the package named 'rpm'.

Читать далее »

Tags: ,
Категория: Linux | Нет комментариев »


How to pack Java Cryptography Extension (JCE) to JDK RPM

Date July 22nd, 2016 Author Vitaly Agapov

I don’t have problem. I am part of problem.

Daniel Keyes «The Minds of Billy Milligan»

jce

Let's imagine you have hundreds of hosts where you automatically install or update HotSpot JDK from the RPM-repository. For example, by RPM dependencies while installing or updating some another software. You maintain your repository by yourself and you'd like to have JDK and JCE in one RPM package.

Then this tiny article will help you.

Читать далее »

Tags:
Категория: Linux | 2 Комментариев »


How to query MS SQL Server database from Linux CLI

Date June 15th, 2016 Author Vitaly Agapov

We have no need of other worlds. We need mirrors. We don't know what to do with other worlds.

Stanisław Lem «Solaris»

SQLServerLinuxSo let's assume that we need a small keyhole to look into SQL Server database and query some data from it. Maybe we'd like to automate some reporting stuff or to make some web GUI with some information stored in MSSQL. And of course we don't want to deal with Windows.

Fortunately there is such a project as FreeTDS. It gives us a driver for using with ODBC engine. I will show how to configure it and how to retreive some data right from MSSQL. 

Everything written below is true for the case of RHEL/CentOS 7.

Читать далее »

Tags:
Категория: Linux | Нет комментариев »


How to migrate XWIki and DokiWiki to Confluence: still easy and for free

Date May 11th, 2016 Author Vitaly Agapov

Nothing had ever been so welcome by its absence.

Dan Simmons «The Rise of Endymion»

xwiki-dokuwiki-confluenceI am surely aware of the UWC project but I just wasn't satisfied with the results. So I have created my own tool to migrate articles from XWiki and Dokuwiki to Confluence. The tool is quite simple, it doesn't deal with editing history and just migrates the current information. This was my goal and it was achieved. So if you are interested in this – look into the project WikiLumper which is available on GitHub. 

And if you have something to contribute then please contribute.

Tags: ,
Категория: Perl | Нет комментариев »