Streamlining Network Connectivity Validation in Enterprise Environments

Pain Area

In many enterprises, SOC compliance dictates that the engineers who develop code should not be the ones who deploy it. This segregation of duties is essential for maintaining security and compliance, but it introduces significant inefficiencies, especially when it comes to network connectivity and firewall requests.

Typically, developers need to open network connectivity or firewall requests but lack the necessary permissions to validate these connections. Instead, they rely on DevOps, SRE, or operations teams to test and validate the connections once the firewall or network team has opened them. This process is fraught with delays and inefficiencies:

• Coordination Challenges: Developers must coordinate with multiple teams, each with its own schedule and priorities.
• Wasted Time: Operations teams often find themselves waiting idly for the firewall team to complete their tasks before they can perform their validations.
• Productivity Loss: The constant back-and-forth reduces overall productivity, as both developers and operations teams spend more time on administrative coordination than on actual development and operations work.

Task

The goal was to create a solution that empowers developers to independently validate network connections without breaching SOC compliance guidelines or relying on operations teams. The solution needed to:

• Reduce Dependency: Eliminate the need for operations teams to perform routine connection validations.
• Increase Efficiency: Enable developers to quickly and independently verify network connectivity.
• Enhance Productivity: Allow all teams to focus on their core responsibilities without unnecessary delays.

Action

To address this pain point, I developed a code-based solution that automates the network connectivity validation process. This solution utilizes the code available in the tcpcheck repository on GitHub and leverages a ready-to-use Docker image from Docker Hub. The key steps involved in the solution are:

1. Deploy Docker Container: Use the tcpcheck Docker image to deploy a container within the enterprise network. This container provides an internal DNS URL that can be accessed by anyone within the network.
2. HTTP Call Functionality: Allow any user within the network to make an HTTP call to the internal DNS URL, passing the desired DNS and port number to check TCP or HTTPS connections.
3. Automated Validation: The script within the Docker container automatically performs the connection validation and returns the results, indicating whether the connection is successful or not.

The tcpcheck tool is designed to be straightforward and easy to use. By integrating it into the internal DNS URL, it enables seamless and efficient network connectivity checks. Here’s how to use the Docker image:

codedocker pull jayeshmahajan/tcpcheck:latest
docker run -d -p 8080:8080 jayeshmahajan/tcpcheck

Once the container is running, you can check a connection by making an HTTP request to the internal DNS URL. For example:

curl http://<internal_dns_url>:8080/check?host=<target_host>&port=<target_port>

curl 'http://<internal_dns_url/check_http_connection?domain=www.google.com'


    "message": "HTTP connection successful",
    "dns_result": {
        "cname": null,
        "ips": [
            "142.250.81.238"
        ]
    }
}=============================
http://<internal_dns_url/check_http_connection?protocol=https&domain=wrong.host.badssl.com
{
    "message": "SSL connection failed",
    "dns_result": {
        "cname": null,
        "ips": [
            "104.154.89.105"
        ]
    },
    "error": "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'wrong.host.badssl.com'. (_ssl.c:1133)"
}=============================
http://<internal_dns_url/check_http_connection?protocol=https&domain=www.ge.com
{
    "message": "HTTP connection successful",
    "dns_result": {
        "cname": "www.ge.com.cdn.cloudflare.net.",
        "ips": [
            "104.18.31.171",
            "104.18.30.171"
        ]
    }
}

Result

The implementation of this solution yielded significant benefits:

• Reduced Dependency on Operations Teams: Developers can now independently validate network connections, eliminating the need for constant coordination with operations teams.
• Faster Feature Delivery: By empowering developers to perform their own validations, the solution speeds up the development and deployment process, allowing features to be delivered more quickly.
• Improved Productivity: Both developers and operations teams can now focus on their primary tasks, reducing downtime and increasing overall productivity.
• Enhanced Responsiveness During Outages: In the event of a network outage, any team member, including leadership, can quickly validate network connections without waiting for the operations team, leading to faster issue resolution.

This innovative solution has proven to be particularly useful during network outages, where time is of the essence. By removing unnecessary dependencies and streamlining the validation process, it ensures that everyone in the organization can contribute to maintaining a robust and reliable network infrastructure.

---

This approach not only enhances operational efficiency but also aligns with SOC compliance requirements, demonstrating that security and productivity can go hand in hand with the right solutions in place. The tcpcheck tool and its Docker image exemplify how a simple yet effective solution can make a significant impact on enterprise operations.

Cloud is the Limit: Google Cloud Platform Database Services
GCP offers several database services that you can choose from.

Cloud SQL:
A relational GCP database service that is fully managed and compatible with MySQL, PostgreSQL and SQL Server, Cloud SQL includes features like automated backups, data replication, and disaster recovery to ensure high availability and flexibility.

When to choose: From ‘lift and shift’ of on-premise SQL databases to the cloud to handling large-scale SQL data analytics to supporting CMS data storage and scalability and deployment of micro services, Cloud SQL has many uses and is a better option when you need relational database capabilities but don’t need storage capacity over 10TB.

Cloud Spanner:
Another fully managed, relational Google Cloud database service, Cloud Spanner differs from Cloud SQL by focusing on combining the benefits of relational structure and non-relational scalability. It provides consistency across rows and high-performance operations and includes features like built-in security, automatic replication, and multi-language support.

When to choose: Cloud Spanner should be your go-to option if you plan on using large amounts of data (more than 10TB) and need transactional consistency. It is also a perfect choice if you wish to use sharding for higher throughput and accessibility.

BigQuery:
With BigQuery you can perform data analyses via SQL and query stream-data. Since BigQuery is a serverless data warehouse that’s fully managed, its built-in Data Transfer Service helps you migrate data from on-premises resources, including Teradata.

It incorporates features for machine learning, business intelligence, and geospatial analysis that are provided through BigQuery ML, BI Engine, and GIS.

When to choose: Use cases for BigQuery involve process analytics and optimization, big data processing and analytics, data warehouse modernisation, machine learning-based behavioural analytics and predictions.

Cloud Bigtable:
It is a fully managed NoSQL Google Cloud database service that is designed for large operational and analytics workloads. Cloud Bigtable includes features for high availability and zero-downtime configuration changes. You can practically integrate it with a variety of tools, including Apache tools and Google Cloud services.

Cloud Bigtable use cases cover financial analysis and prediction, IoT data ingestion, processing, and analytics, and hyper-personalised marketing applications.

When to choose: Cloud Bigtable is a good option if you are using large amounts of single key data and is preferable for low-latency, high throughput workloads.

Cloud Firestore:
A fully managed, serverless NoSQL GCP database designed for the development of serverless apps, Cloud Firestore can be used to store, sync, and query data for web, mobile, and IoT applications. With critical features like offline support, live synchronization, and built-in security, you can even integrate Firestore with Firebase, GCP’s mobile development platform, for easier app creation.

Cloud Firestore use cases include mobile and web applications with both online and offline capabilities, multi-user, collaborative applications, real-time analytics, social media applications, and gaming forums and leaderboards.

When to choose: When your focus lies on app development and you need live synchronization and offline support.

Firebase Realtime Database:
This is a NoSQL Google Cloud database that is a part of the Firebase platform. It allows you to store and sync data in real-time and includes caching capabilities for offline use. It also enables you to implement declarative authentication, matching users by identity or pattern.

It includes mobile and web software development kits for easier app development.
Use cases for Firebase Realtime Database involve development of apps that work across devices, advertisement optimisation and personalisation, and third-party payment processing.

Cloud Memorystore:
Designed to be secure, highly available, and scalable, Cloud Memorystore is a fully managed, in-memory Google Cloud data store that enables you to create application caches with sub-millisecond latency for data access.

Use cases for Cloud Memorystore include ‘lift and shift’ migration of applications, machine learning applications, real-time analytics, low latency data caching and retrieval.

When to choose: If you are using key-value datasets and your main focus is transaction latency.

Choosing the database on key questions
I also created this flowchart that can show a direction in terms of selecting the database:

Fish ~/.config/fish/config.fish Example
set PATH $PATH /usr/local/bin
export PATH
function ll
ls -lh $argv
end
set -g PATH $PATH /Applications/Visual\ Studio\ Code.app/Contents/Resources/app/bin

Problem statement:

When you run docker container, lots of enterprise organization do not allow you to run container as root or as sudo because it compromises the container file system access and for some other various reason.

I run into same situation while that I wanted to run the process as a user but my virtual machine instance username doesn’t match with container’s username’s UID.

e.g.

$ id
uid=1000(circleci) gid=1000(circleci) groups=1000(circleci),4(adm),20(dialout),24(cdrom),25(floppy),29(audio),30(dip),44(video),46(plugdev),117(netdev),118(lxd),997(docker)

$ docker run -it --rm cimg/node:$CIRCLECI_NODE_TAG id
uid=3031(circleci) gid=3031(circleci) groups=3031(circleci),4(adm),20(dialout),24(cdrom),25(floppy),29(audio),30(dip),44(video),46(plugdev),117(netdev),118(lxd),997(docker)

so when you mount the file and run as circleci user it would fail.

Resolution:

If you want to run this as same uid as what it has on VM

print(“Hello, World!”)
x = 5
y = “John”
print(x)
print(y)
x = int(1) # x will be 1
x = float(1) # x will be 1.0
a = ” Hello, World! “
print(a.strip())
print(a[1])
print(a.split(“,”))
print(a.replace(“a”,”b”))
thislist = [“apple”, “banana”, “cherry”]
print(thislist)
print(thislist[1])
thislist.append(“orange”)
thislist.insert(1, “orange”)
thislist.remove(“banana”)
thislist.pop()
del thislist[0]
thislist.clear()
mylist = thislist.copy()
thistuple = (“apple”, “banana”, “cherry”)
print(thistuple)
if “apple” in thistuple:
print(“Yes, ‘apple’ is in the fruits tuple”)
thisset = {“apple”, “banana”, “cherry”}
thisset.add(“orange”)
print(thisset)
thisset.update([“orange”, “mango”, “grapes”])
print(thisset)
thisset.discard(“banana”)
thisdict = {
“brand”: “Ford”,
“model”: “Mustang”,
“year”: 1964
}
print(thisdict)
x = thisdict[“model”]
thisdict[“year”] = 2018
for x in thisdict:
print(x)
for x in thisdict:
print(dict[x])
for x in thisdict.values():
print(x)
for x, y in thisdict.items():
print(x, y)
if “model” in thisdict:
print(“Yes, ‘model’ is one of the keys in the thisdict dictionary”)
thisdict.pop(“model”)
if b > a:
print(“b is greater than a”)
elif a == b:
print(“a and b are equal”)
else:
print(“a is greater than b”)
i = 1
while i < 6:
print(i)
i += 1
for x in range(2, 6):
print(x)
for x in range(2, 30, 3):
print(x)
def my_function():
print(“Hello from a function”)
try:
print(x)
except:
print(“An exception occurred”)
try:
print(x)
except NameError:
print(“Variable x is not defined”)
except:
print(“Something else went wrong”)
f = open(“demofile.txt”)
f = open(“demofile.txt”, “r”)
print(f.read())
print(f.readline())
f.close()
f = open(“demofile2.txt”, “a”)
f.write(“Now the file has more content!”)
f.close()
f = open(“myfile.txt”, “w”) // create new if it doesnt exist
f = open(“myfile.txt”, “x”) //create new
import os
os.remove(“demofile.txt”)

print(“Hello, World!”)

x = 5
y = “John”
print(x)
print(y)

x = int(1) # x will be 1

x = float(1) # x will be 1.0

a = ” Hello, World! “
print(a.strip())

print(a[1])

print(a.split(“,”))

print(a.replace(“a”,”b”))

thislist = [“apple”, “banana”, “cherry”]
print(thislist)
print(thislist[1])

thislist.append(“orange”)

thislist.insert(1, “orange”)

thislist.remove(“banana”)

thislist.pop()

del thislist[0]

thislist.clear()

mylist = thislist.copy()

thistuple = (“apple”, “banana”, “cherry”)
print(thistuple)

if “apple” in thistuple:
print(“Yes, ‘apple’ is in the fruits tuple”)

thisset = {“apple”, “banana”, “cherry”}

thisset.add(“orange”)

print(thisset)

thisset.update([“orange”, “mango”, “grapes”])

print(thisset)

thisset.discard(“banana”)

thisdict = {
“brand”: “Ford”,
“model”: “Mustang”,
“year”: 1964
}
print(thisdict)

x = thisdict[“model”]

thisdict[“year”] = 2018

for x in thisdict:
print(x)

for x in thisdict:
print(dict[x])

for x in thisdict.values():
print(x)

for x, y in thisdict.items():
print(x, y)

if “model” in thisdict:
print(“Yes, ‘model’ is one of the keys in the thisdict dictionary”)

thisdict.pop(“model”)

if b > a:
print(“b is greater than a”)

elif a == b:
print(“a and b are equal”)

else:
print(“a is greater than b”)

i = 1
while i < 6:
print(i)
i += 1

for x in range(2, 6):
print(x)

for x in range(2, 30, 3):
print(x)

def my_function():
print(“Hello from a function”)

try:
print(x)
except:
print(“An exception occurred”)

try:
print(x)
except NameError:
print(“Variable x is not defined”)
except:
print(“Something else went wrong”)

f = open(“demofile.txt”)

f = open(“demofile.txt”, “r”)
print(f.read())

print(f.readline())

f.close()

f = open(“demofile2.txt”, “a”)
f.write(“Now the file has more content!”)
f.close()

f = open(“myfile.txt”, “w”) // create new if it doesnt exist

f = open(“myfile.txt”, “x”) //create new

import os
os.remove(“demofile.txt”)

terraform init -input=false -backend=true -backend-config="bucket=${WHATEVER_S3_BUCKET}" -backend-config="key=state/terraform.tfstate" -backend-config="region=us-east-1" -backend-config="profile=${WHATEVER_PROFILE}"


terraform plan -var-file=tfvars/${ENV}.tfvars -out tf.out


terraform apply "tf.out" #  -auto-approve

gpg –list-keys
this gives you a list of all the keys on your computer. you need this to find the keyname that you are trying to update.
## name_of_the_key=`gpg –list-keys |grep -i Jayesh |grep -i uid |awk ‘{print $4}’`
gpg –edit-key [name_of_the_key]
command> list
lists the available subkeys
command> key [subkey]
choose the number of the subkey you want to edit; e.g. key 1
command> expire
expire lets you set a new expiration date for the subkey.
command> save

helm install ./ –debug –dry-run –generate-name -f values1.yaml

Additional options:

--namespace <string>

helm template –debug