Solving Known Host Quirk

Technical Background

When a server is re-created, its SSH host key changes, triggering warnings like:

ssh root@<your-server-ip>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

This prevents automated pipelines from connecting without manual confirmation.
The solution is to generate a deterministic known_hosts file containing the current server’s SSH public key and to use wrapper scripts for ssh and scp. These wrappers instruct the SSH client to use this controlled known_hosts file instead of the global ~/.ssh/known_hosts.

Info

This approach is especially useful in CI/CD pipelines or Terraform-managed lab environments where servers are frequently destroyed and recreated.

Solution

Prerequisits

Create a main.tf, outputs.tf, /tpl/userData.yaml variables.tf, network.tf,providers.tf and secrets.auto.tfvars like in 13 Working On Cloud Init.

SSH Key for Server

  1. Add a key pair resource (if not already present):
resource "tls_private_key" "host" {
  algorithm = "ED25519"
}
  1. Modify userData.yml to add new keys by adding to your local_file ressource:
resource "local_file" "user_data" {
  content = templatefile("tpl/userData.yml", {
    host_ed25519_private = indent(4, tls_private_key.host.private_key_openssh)
    host_ed25519_public  = tls_private_key.host.public_key_openssh
    devopsSSHPublicKey   = hcloud_ssh_key.loginUser.public_key
  })
  filename = "gen/userData.yml"
}

Info

This ensures each server has a unique host key instead of using image defaults.

Generate known_hosts file

Add a local_file ressource to your main.tf containing:

resource "local_file" "known_hosts" {
  content         = "${hcloud_server.web.ipv4_address} ${tls_private_key.host.public_key_openssh}"
  filename        = "gen/known_hosts"
  file_permission = "644"
}

Warning

Always use the public key (public_key_openssh), not the fingerprint. Using a fingerprint will result in invalid known_hosts format.

Creating a SSH Wrapper Template

  1. Create a file tpl/ssh.sh containing:
#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh \
  -o UserKnownHostsFile="$GEN_DIR/known_hosts" \
  -o IdentitiesOnly=yes \
  -i ~/.ssh/id_ed25519 \
  ${devopsName}@${ip}
  1. Generate the wrapper with a ressource in your main.tf containing:
resource "local_file" "ssh_script" {
  content = templatefile("tpl/ssh.sh", {
    ip = "${hcloud_server.web.ipv4_address}"
    devopsName = "${var.loginUser_name}"
  })
  filename        = "gen/ssh.sh"
  file_permission = "700"
  depends_on      = [local_file.known_hosts]
}

Creating SCP Wrapper Template

  1. Create a file tpl/scp.sh containing:
#!/usr/bin/env bash
GEN_DIR=$(dirname "$0")/../gen

if [ $# -lt 2 ]; then
    echo "Usage: $0 <source> <destination>"
    exit 1
fi

scp -o UserKnownHostsFile="$GEN_DIR/known_hosts" "$@"
  1. Generate the wrapper with a ressource in your main.tf containing:
resource "local_file" "scp_script" {
  content         = templatefile("${path.module}/tpl/scp.sh", {})
  filename        = "${path.module}/bin/scp"
  file_permission = "755"
  depends_on      = [local_file.known_hosts]
}

Usage

  1. Generate files using Terraform
terraform init
terraform apply

Info

This generates the known_host and userData.yaml with the actual content inside /gen. Inside /gen files get stored that are typically not executed directly but are used as input or data for other processes.

This generates the ssh.sh and scp.sh with the actual content inside bin. Inside /bin files get stored that are executable scripts or utilities that you run manually or via automation.

  1. Connect to your server and execute:
./bin/ssh
./bin/scp localfile devops@<your-server-ip>:~

Info

This runs the scripts and transfers the correct files. They ensure the correct known_hosts file is always used, eliminating host identification changed warnings.

Wrapper scripts bypass the default ~/.ssh/known_hosts, so other host keys stored there are ignored.

Warning

Ensure the scripts in bin/ are executable (chmod +x bin/ssh bin/scp).

If the server IP changes but you don’t re-run terraform apply, the known_hosts entry will be outdated.

cloud-init docs

Terraform Modules