Author: sskaje

  • Proxmox VE 创建集群

    空的集群不提了。家里有三台电脑,都装了PVE,都有虚拟机,直接创建集群将三个节点都加入,从Web console操作是不行的。命令行操作步骤如下:

    1. 修改主机名

    确保每台pve主机的主机名不一样。

    假定原来主机名叫 pve,新的主机名叫 pve1,我都加了 local 的后缀。

    hostnamectl set-hostname pve1.home.local

    然后修改 /etc/hosts/etc/postfix/main.cf,确保里边的主机名是新的。

    但是这样操作是不够的,还得去手动处理主机配置文件。(打开web console,看到的节点变成了两个,pve和pve1)

    mv /etc/pve/nodes/pve/qemu-server/* /etc/pve/nodes/pve1/qemu-server/

    建议顺手更新,然后重启系统。

    2. 删除已有的集群信息(可选)

    如果之前尝试过建集群,需要删掉这个集群,否则后边会进不了cluster的页面。

    文档: https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

    systemctl stop pve-cluster corosync
    pmxcfs -l
    rm /etc/corosync/*
    rm /etc/pve/corosync.conf
    killall pmxcfs
    systemctl start pve-cluster

    3. 修改主机ID

    我其实不确定是不是必须,反正我是改了

    脚本参考:https://forum.proxmox.com/threads/changing-vmid-of-a-vm.63161

    #!/usr/bin/env bash
    
    VM_TYPE=$1
    OLD_VMID=$2
    NEW_VMID=$3
    
    #echo Put the VM type to change '(lxc, qemu)'
    #read -r VM_TYPE
    case "$VM_TYPE" in
    "lxc") VM_TYPE="lxc" ;;
    "qemu" | "qemu-server")
    VM_TYPE="qemu-server" ;;
    *)
      echo bad input. Exiting
    exit
      ;;
    esac
    
    #echo
    #echo Put the VMID to change
    #read -r OLD_VMID
    case $OLD_VMID in
    '' | *[!0-9]*)
      echo bad input. Exiting
    exit
      ;;
    *)
      echo Old VMID - "$OLD_VMID"
      ;;
    esac
    
    #echo
    #echo Put the new VMID
    #read -r NEW_VMID
    case $NEW_VMID in
    '' | *[!0-9]*)
      echo bad input. Exiting
    exit
      ;;
    *)
      echo New VMID - "$NEW_VMID"
      ;;
    esac
    echo
    
    VG_NAME="$(lvs --noheadings -o lv_name,vg_name | grep "$OLD_VMID" | awk -F ' ' '{print $2}' | uniq)"
    
    case "$VG_NAME" in
    "")
      echo Machine not in Volume Group. Exiting
      exit
      ;;
    *)
      echo Volume Group - "$VG_NAME"
      ;;
    esac
    
    for volume in $(lvs -a | grep "$VG_NAME" | awk '{print $1}' | grep "$OLD_VMID"); do
      newVolume="${volume//"${OLD_VMID}"/"${NEW_VMID}"}"
      lvrename "$VG_NAME" "$volume" "$newVolume"
    done
    
    for volume in $(zfs list -t all | awk '{print $1}' | grep "vm-${OLD_VMID}-disk"); do
      newVolume="${volume//"${OLD_VMID}"/"${NEW_VMID}"}"
      zfs rename "$volume" "$newVolume"
    done
    
    sed -i "s/$OLD_VMID/$NEW_VMID/g" /etc/pve/"$VM_TYPE"/"$OLD_VMID".conf
    mv /etc/pve/"$VM_TYPE"/"$OLD_VMID".conf /etc/pve/"$VM_TYPE"/"$NEW_VMID".conf
    

    原脚本是交互输入参数的,我改成命令行参数了,这样可以搭配 for 去批量改id。

    我的pve vm都是lvm的,所以这样没问题。如果是独立磁盘文件的,需要rename,然后qm rescan

    4. 移动主机文件到一个统一的节点

    假设我用 pve1 当主节点(我不知道pve有没有主节点的概念,就当临时用来操作的),pve1自己不动,其他节点需要把主机文件scp过去

    假定pve1 ip是 192.168.111.1,现在处理 pve2

    scp -pr /etc/pve/nodes/pve2 root@192.168.111.1:/etc/pve/nodes/
    mv /etc/pve/nodes/pve2 /root/

    5. 创建并加入集群

    从pve1 创建集群,从其他已经处理好的节点加集群。

    完。

  • pve ubuntu cloud-init

    自己装的ubuntu,转成模板后,增加cloud-init支持。

    root 下

    truncate -s0 /etc/hostname
    hostnamectl set-hostname localhost
    
    rm -f /etc/netplan/50-cloud-init.yaml
    
    
    truncate -s0 /etc/machine-id
    
    
    cloud-init clean
    
    
    truncate -s0 ~/.bash_history
    
    history -c
    
    shutdown -h now
    

    关机后,hardware 增加 cloud-init drive。

    修改 cloud-init 设置。

    ubuntu 2404验证通过。

  • Proxmox VE 恢复 root SSH 登录

    Proxmox VE 的各种说明里都是让进单用户模式去改密码,之前试过几个古老版本,live usb载入 + chroot + passwd 改出来的root密码都无效,不知道原因。

    现在换另外一个方案,增加 ssh authorized_keys。

    在 pve里,authorized_keys 使用的是symbolic link,链接到了 /etc/pve/priv/authorized_keys

    # ls -al root/.ssh/
    total 24
    drwxr-xr-x 2 root root 4096 Mar 24  2021 .
    drwx------ 3 root root 4096 Dec 11  2020 ..
    lrwxrwxrwx 1 root root   29 Dec 11  2020 authorized_keys -> /etc/pve/priv/authorized_keys
    

    根据 PVE 的wiki (https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)#_recovery) ,/etc/pve 使用的是他家的 pmxcfs 文件系统,数据实际存储在 /var/lib/pve-cluster/config.db

    file 一下,可以看出是 sqlite3,打开,只有一张表 tree,定义如下:

    CREATE TABLE tree (  inode INTEGER PRIMARY KEY NOT NULL,  parent INTEGER NOT NULL CHECK(typeof(parent)=='integer'),  version INTEGER NOT NULL CHECK(typeof(version)=='integer'),  writer INTEGER NOT NULL CHECK(typeof(writer)=='integer'),  mtime INTEGER NOT NULL CHECK(typeof(mtime)=='integer'),  type INTEGER NOT NULL CHECK(typeof(type)=='integer'),  name TEXT NOT NULL,  data BLOB);
    

    特别简单。

    改数据就直接 update 表。

  • Proxmox VE 虚拟机磁盘挂载


    QEMU QCOW Image (v3)

    # file vm-101-disk-0.qcow2 
    vm-101-disk-0.qcow2: QEMU QCOW Image (v3), 107374182400 bytes (v3), 107374182400 bytes

    apt install qemu-utils
    
    modprobe nbd max_part=8
    qemu-nbd --connect=/dev/nbd0 vm-1061-disk-0.qcow2
    
    # unload 
    qemu-nbd -d /dev/nbd0
    

    After this

    
    Disk /dev/nbd0: 100 GiB, 107374182400 bytes, 209715200 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x28288fb3
    
    Device      Boot    Start       End   Sectors  Size Id Type
    /dev/nbd0p1 *        2048   3905535   3903488  1.9G 83 Linux
    /dev/nbd0p2       3905536  11718655   7813120  3.7G 82 Linux swap / Solaris
    /dev/nbd0p3      11718656 209715199 197996544 94.4G 83 Linux

    RAW

    vm-1014-disk-1.raw: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,1), end-CHS (0x3ff,254,63), startsector 1, 209715199 sectors, extended partition table (last)
    

    losetup -f -P --show vm-1014-disk-1.raw
    # output like /dev/loop16
    
    # unload 
    losetup -d /dev/loop16

    After this

    Disk /dev/loop16: 100 GiB, 107374182400 bytes, 209715200 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: gpt
    Disk identifier: 49400ED0-2144-4854-BA3D-64A88C3EC9C8
    
    Device        Start       End   Sectors  Size Type
    /dev/loop16p1  2048 209713151 209711104  100G Microsoft basic data
    GPT PMBR size mismatch (1952448511 != 1953525167) will be corrected by write.
    The backup GPT table is not on the end of the device.

    LVM

    From lsblk

    /dev/mapper/pve-vm--118--disk--0

    apt-get install kpartx
    kpartx -a /dev/mapper/pve-vm--118--disk--0
    

    After this,

    # ls /dev/mapper/pve-vm--118--disk--*
    /dev/mapper/pve-vm--118--disk--0  /dev/mapper/pve-vm--118--disk--0p1  /dev/mapper/pve-vm--118--disk--0p2  /dev/mapper/pve-vm--118--disk--0p3
    
    # fdisk -l 
    ...
    
    Disk /dev/mapper/pve-vm--118--disk--0: 50 GiB, 53687091200 bytes, 104857600 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 65536 bytes / 65536 bytes
    Disklabel type: dos
    Disk identifier: 0x28288fb3
    
    Device                             Boot    Start       End  Sectors  Size Id Type
    /dev/mapper/pve-vm--118--disk--0p1 *        2048   3905535  3903488  1.9G 83 Linux
    /dev/mapper/pve-vm--118--disk--0p2       3905536  11718655  7813120  3.7G 82 Linux swap / Solaris
    /dev/mapper/pve-vm--118--disk--0p3      11718656 104855551 93136896 44.4G 83 Linux
    
    # undo
    kpartx -d /dev/mapper/pve-vm--118--disk--0

  • Protected: 海康威视R1开SSH(文章加密,勿扰)

    This content is password-protected. To view it, please enter the password below.

  • Proxmox VE Import VMware VM (VMX)

    1 Download OVFTool on PVE: https://developer.broadcom.com/tools/open-virtualization-format-ovf-tool/latest, extract

    2 Navigate to VMX directory, command like

    /root/ovftool/ovftool myVM.vmx /root/myVM.ova

    3 Extract OVA

    cd /root
    mkdir extracted
    tar -C extracted -xvf myVM.ova

    4 Import VM

    cd extracted
    qm importovf 117 myVM.ovf  local-lvm

    117 is my new VMID

    5 Remove OVA and OVF

    rm -fr /root/myVM.ova /root/extracted

    6 Start VM and install virtio drivers.

  • 海康威视R1关闭外网访问

    关闭几个dns的解析,然后杀进程,就是每次都得手动

    # block mqtt
    chmod -x /histor/bin/emclient
    killall  emclient
    
    # block proxy
    # msgType=110 to api.hiksemi.cn//gateway/api
    # got proxyDataHost/proxyFilePort/proxyServerPort
    chmod -x /histor/bin/network/proxyDaemon
    killall proxyDaemon
  • Install DebugServer for iOS

    Extract DebugServer

    $ ls /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport/
    
    ...
    
    $ hdiutil mount /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport/15.5/DeveloperDiskImage.dmg
    正在检查“Whole_Disk”的校验和…
    ..................................................................................................................................................................................................................................................
                              Whole_Disk:已验证CRC32 $D034FA6C
    已验证CRC32 $8E250F8D
    /dev/disk4          	                               	/Volumes/DeveloperDiskImage
    
    $ cp /Volumes/DeveloperDiskImage/usr/bin/debugserver . 
    $ hdiutil unmount /Volumes/DeveloperDiskImage
    "/Volumes/DeveloperDiskImage" unmounted successfully.
    
    $ file debugserver 
    debugserver: Mach-O universal binary with 2 architectures: [arm64:Mach-O 64-bit executable arm64] [arm64e:Mach-O 64-bit executable arm64e]
    debugserver (for architecture arm64):	Mach-O 64-bit executable arm64
    debugserver (for architecture arm64e):	Mach-O 64-bit executable arm64e
    
    $ xcrun -sdk iphoneos lipo -thin arm64  debugserver -output debugserver_arm64
    

    Entitlements

    View

    codesign -d -v --entitlements - --xml debugserver_arm64 
    Executable=/Users/sskaje/Work/hiksemi/debugserver_arm64
    Identifier=com.apple.debugserver
    Format=Mach-O thin (arm64)
    CodeDirectory v=20400 size=5134 flags=0x2(adhoc) hashes=150+7 location=embedded
    Signature=adhoc
    Info.plist entries=5
    TeamIdentifier=not set
    Sealed Resources=none
    Internal requirements count=0 size=12
    <?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>seatbelt-profiles</key><array><string>debugserver</string></array><key>com.apple.private.cs.debugger</key><true/><key>com.apple.private.memorystatus</key><true/><key>com.apple.security.network.client</key><true/><key>com.apple.security.network.server</key><true/><key>com.apple.private.logging.diagnostic</key><true/><key>com.apple.backboardd.debugapplications</key><true/><key>com.apple.frontboard.debugapplications</key><true/><key>com.apple.backboardd.launchapplications</key><true/><key>com.apple.frontboard.launchapplications</key><true/><key>com.apple.springboard.debugapplications</key><true/></dict></plist>

    ent.xml

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
    	<key>com.apple.private.cs.debugger</key><true/>
    	<key>com.apple.private.memorystatus</key><true/>
    	<key>com.apple.security.network.client</key><true/>
    	<key>com.apple.security.network.server</key><true/>
    	<key>com.apple.private.logging.diagnostic</key><true/>
    	<key>com.apple.backboardd.debugapplications</key><true/>
    	<key>com.apple.frontboard.debugapplications</key><true/>
    	<key>com.apple.backboardd.launchapplications</key><true/>
    	<key>com.apple.frontboard.launchapplications</key><true/>
    	<key>com.apple.springboard.debugapplications</key><true/>
    
    	<key>run-unsigned-code</key> <true/>
    	<key>get-task-allow</key> <true/>
    	<key>task_for_pid-allow</key> <true/>
    
    </dict>
    </plist>
    

    debugserver has entitlements

    	<key>seatbelt-profiles</key>
    	<array>
    	<string>debugserver</string>
    	</array>

    it causes

    iPhone:~ root# debugserver 192.168.11.40:12345 -waitfor HiStor 
    debugserver-@(#)PROGRAM:LLDB  PROJECT:lldb-1316.2.4.18
     for arm64.
    Waiting to attach to process HiStor...
    Listening to port 12345 for a connection from 192.168.11.40...
    Failed to get connection from a remote gdb process.
    Exiting.
    

    Replace Entitlements

    codesign -s - --entitlements ent.xml -f debugserver_arm64 
    debugserver_arm64: replacing existing signature

    Scp to server

    scp debugserver_arm64 root@192.168.11.11:~/

    If on palera1n

    cp debugserver_arm64 /var/jb/usr/bin/debugserver

    HiStor

    $ debugserver '*:12345' -waitfor HiStor

    lldb

    (lldb) platform select remote-ios
      Platform: remote-ios
     Connected: no
      SDK Path: "/Users/sskaje/Library/Developer/Xcode/iOS DeviceSupport/iPhone8,1 15.8.2 (19H384)"
     SDK Roots: [ 0] "/Users/sskaje/Library/Developer/Xcode/iOS DeviceSupport/iPhone8,1 15.8.2 (19H384)"
    (lldb) target create HiStor
    Current executable set to '/Users/sskaje/Work/hiksemi/ios/Payload/HiStor.app/HiStor' (arm64).
    (lldb) process connect connect://192.168.11.156:12345
    

    Useless note

    (lldb) memory read -f y -c 4 -s 1 $pc
    0x105280e98: 48 61 74 94
    (lldb) memory write  $pc f0 60 74 94 
    (lldb) memory read -f y -c 4 -s 1 $pc
    0x105280e98: f0 60 74 94
    

  • 海康威视 R1 开SSH的探索

    618买了个海康威视R1,N100 + 8G内存,配置很好,但是看评价说系统很差,我想想再差还能差到哪儿,等了一周多到货了,终于发现,买了一个NAS却感觉设备不是自己的,体验很糟糕。

    不想刷黑群晖,所以这个方向不讨论了。

    从数据安全角度出发,我需要让设备可以被禁止访问外网,或者限定访问目标和内容,防止数据泄漏。于是设备跑了两天多,我只敢把之前下载的电影动漫放上去,其他资料一个都不敢放,白瞎了4*8T。

    已知信息,N100,网上有黑群晖教程并提及优先USB引导。所以搞了个Debian的live usb,启动,点点按按看到了bios的提示,也进了Debian。

    系统是装在一个32GB的flash上的,基于debian改造的一个系统。

    之前劫持NAS的互联网流量时,抓到了固件更新的接口,但是格式应该是自定义格式,里边能提取一个 histor.img,但是应该是加密或者其他处理过的,没法用。

    Debian进去后,逐个分区看,找到了一个 histor.img 和一个 sq.img。整盘dd出来,从 linux 里直接挂载分区,并squashfuse 加载img,一切都正常。期间涉及到的命令为:

    # from https://askubuntu.com/a/998269
    # 挂载
    losetup -f -P hiksemi-r1.img
    # 看所有loop
    losetup -l
    # 找到这个设备后mount

    两个img,一个是root fs,一个是海康自己的软件,尝试用web 的一些出错,找到了几个文件,里边都加载了完整的server 的逻辑,所以挑了一个出来反编译。而在其他文件里发现了一些跟ssh,跟调试,跟密码有关的脚本或者配置文件。

    目前还在进行中,有部分进展不适合发出来,后边再看怎么写

    UPDATE 20240812

    有人来问进展,搞定了,但是没计划公开。分析过程比较复杂,提root过程用了一些奇怪的技巧。配置没法持久化,每次重启都需要重新开,不必浪费时间了,有需求的请刷其他系统。

  • OpenWRT x86 + Quectel 5G Module

    # On PVE Host

    Create a Linux VM without disk, in my PVE, the new VM id is 115.

    wget https://openwrt.proxy.ustclug.org/snapshots/targets/x86/64/openwrt-x86-64-generic-ext4-combined.img.gz
    gunzip openwrt-x86-64-generic-ext4-combined.img.gz
    qemu-img resize -f raw openwrt-x86-64-generic-ext4-combined.img 20G
    qm importdisk 115 openwrt-x86-64-generic-ext4-combined.img local-lvm
    rm -f openwrt-x86-64-generic-ext4-combined.img

    From VM Hardware, click the newly imported disk, open the dialog and save.

    Then Change boot order, add the disk.

    Start OpenWRT VM

    # On OpenWRT

    uci set network.lan.ipaddr='192.168.11.243'
    uci commit
    sed -i -e 's/downloads.openwrt.org/openwrt.proxy.ustclug.org/g' /etc/opkg/distfeeds.conf
    reboot
    
    # Now you should be able to access VM from other machines.
    
    # temporary internet access
    ip route add default via 192.168.11.1
    echo 'nameserver 192.168.11.1' > /etc/resolv.conf
    
    opkg update
    
    # install web ui
    opkg install luci bash mtr
    
    # change default shell to bash
    sed -i -e 's/bin\/ash/bin\/bash/g' /etc/passwd
    
    # install wwan module (NCM mode) support
    opkg install kmod-usb-net-rndis kmod-usb-acm luci-proto-ncm picocom
    
    # Resize disk
    # https://openwrt.org/docs/guide-user/advanced/expand_root
    opkg install parted losetup resize2fs
    wget -U "" -O expand-root.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/expand_root?codeblock=0"
    . ./expand-root.sh
    
    reboot

    # On PVE Console

    Load Quectel Module by Adding USB Hardware

    # On OpenWRT Console

    Network > Interfaces > Add New Interface

    Name: wwan

    Protocol: DHCP client

    Device: usb0

    You should be able to find usb0 by `ip a`

    Firewall, set Zone to WAN.

    Save & Apply

    # Note

    DO NOT ATTACH Quectel WWAN module before OpenWRT fully booted.